I’ve been working on getting sqlite running on an embedded system.

In this case it’s a stm32f429-discovery board, running Chibios.

A couple of notes:

1) I trimmed down a lot, my compile options:

-DSQLITE_OMIT_WAL -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_OMIT_AUTHORIZATION -DSQLITE_OMIT_EXPLAIN -DSQLITE_THREADSAFE=0 -DSQLITE_TEMP_STORE=3 -DSQLITE_ENABLE_MEMSYS5

2) I had to define my own heap. This board has 8M of ram, so I was running all memory databases off of this heap.
In order do do this, I had to shutdown whatever the amalgamation started up, re-configure and then re-initialize.

sqlite3_shutdown();
rc = sqlite3_config(SQLITE_CONFIG_HEAP,SDRAM_BASE,SDRAM_SIZE,512);
sqlite3_initialize();

3) I had to bump my stack size in the Makefile:
# Stack size to be allocated to the Cortex-M process stack. This stack is
# the stack used by the main() thread.
ifeq ($(USE_PROCESS_STACKSIZE),)
USE_PROCESS_STACKSIZE = 0x1000
endif

4) I had to remove all of the unix vfs plugins from the sqlite3.c file – this was the only code mod I’ve made to the actual amalgamation file. I had to remove all of the unix vfs to get around the system calls that they made that I didn’t have due to the small environment that I am running in.

/*
** All default VFSes for unix are contained in the following array.
**
** Note that the sqlite3_vfs.pNext field of the VFS object is modified
** by the SQLite core when the VFS is registered. So the following
** array cannot be const.
*/
static sqlite3_vfs aVfs[] = {

};
/*
#if SQLITE_ENABLE_LOCKING_STYLE && defined(__APPLE__)
UNIXVFS("unix", autolockIoFinder ),
#elif OS_VXWORKS
UNIXVFS("unix", vxworksIoFinder ),
#else
UNIXVFS("unix", posixIoFinder ),
#endif
UNIXVFS("unix-none", nolockIoFinder ),
UNIXVFS("unix-dotfile", dotlockIoFinder ),
UNIXVFS("unix-excl", posixIoFinder ),
#if OS_VXWORKS
UNIXVFS("unix-namedsem", semIoFinder ),
#endif
#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
UNIXVFS("unix-posix", posixIoFinder ),
#endif
#if SQLITE_ENABLE_LOCKING_STYLE
UNIXVFS("unix-flock", flockIoFinder ),
#endif
#if SQLITE_ENABLE_LOCKING_STYLE && defined(__APPLE__)
UNIXVFS("unix-afp", afpIoFinder ),
UNIXVFS("unix-nfs", nfsIoFinder ),
UNIXVFS("unix-proxy", proxyIoFinder ),
#endif
};*/

5) I had to define my own VFS and register it- (sqlite won’t start up without a default VFS) – I took a copy of test_demovfs.c that was on the sqlite site, I removed all actual system calls (i.e. fopen, write, etc) so it became a do-nothing vfs – it would fail if I tried to use it. This dosn’t matter because I am using all memory databases for now. Later, I will need to make a vfs that will go to a specific spot in memory where I have some flash attached. – All I have is ram at the moment.

6) use sqlite like normal (I stole this code from a post somewhere):

rc = sqlite3_open(":memory:",&db);

char *sql = "DROP TABLE IF EXISTS Cars;"
"CREATE TABLE Cars(Id INT, Name TEXT, Price INT);"
"INSERT INTO Cars VALUES(1, 'Audi', 52642);"
"INSERT INTO Cars VALUES(2, 'Mercedes', 57127);"
"INSERT INTO Cars VALUES(3, 'Skoda', 9000);"
"INSERT INTO Cars VALUES(4, 'Volvo', 29000);"
"INSERT INTO Cars VALUES(5, 'Bentley', 350000);"
"INSERT INTO Cars VALUES(6, 'Citroen', 21000);"
"INSERT INTO Cars VALUES(7, 'Hummer', 41400);"
"INSERT INTO Cars VALUES(8, 'Volkswagen', 21600);";

rc = sqlite3_exec(db, sql, 0, 0, &err_msg);

...

rc = sqlite3_prepare_v2(db, "SELECT * FROM Cars", -1, &res, 0);

while (sqlite3_step(res) == SQLITE_ROW) {
printf("%s | %s\r\n", sqlite3_column_text(res, 0),
sqlite3_column_text(res, 1));
}

...

and . . .

1 | Audi
2 | Mercedes
3 | Skoda
4 | Volvo
5 | Bentley
6 | Citroen
7 | Hummer
8 | Volkswagen

So this looks promising – I haven’t tested for much of anything yet (i.e. speed, integrity) – but just this basic sanity check looks promising.

Advertisement

Yes, CCL works on Intell Edison. ’nuff said.

PDF version

    In order to understand an Extended Kalman Filter (EKF) I decided to write a simple one that I could test in code. Most of my actual coding so far has been in octave, but eventually it will need to run on an embedded processor.

    My first attempt at a Kalman Filter (KF) came in the context of the Udacity course CS373. These were interesting, but I didn’t feel as if I understood them really well. I am currently going through the book Probabalistic Robotics by Sebastian Thrun, et al. – which I have found to be easy to understand when compared to Optimal State Estimation by Dan Simon, or  Optimal Control and Estimation by Robert Strengel. I was able to create a KF that did a pretty good job of figuring things out, but I still didn’t really understand how EKF’s worked for a non-linear system.
    In order to learn about this a bit better, I gave myself the task of having an EKF figure out a,b and c when my ‘measured’ x and y were of the form y = a\,x^2 + b\,x + c. I started with simple code that generated a series of points at regular intervals of x that fit the quadratic equation, plus gaussian noise in both the x and y directions.

    The beginning place for all of the books on KF’s of all varieties is in math, but assuming a good matrix library, we can almost reduce it to black box. If we get the algorithm set up, and understand what we need to initialize our variables as, we don’t have to struggle through the therory so much. What I am mostly interested here is helping myself to remember what I did, but perahps it will be a useful cross reference for someone else. Let’s start with the algorithm.
    Rephrasing the algorithm in  Probabalistic Robotics Figure 3.3 for the EKF, ignoring u the input vector and process noise, give us:
    \boldsymbol{\bar{x}_{t}}=\boldsymbol{f}(\boldsymbol{\vec{x}_{t-1}},\delta t)
    \boldsymbol{\bar{P}}=\boldsymbol{F\:P_{k-1}\:F^t}
    \boldsymbol{K}=\boldsymbol{\bar{P}\:H^t\:(H\:\bar{P}\:H^t + R)}^{-1}
    \boldsymbol{x_{t}}=\boldsymbol{\bar{x}_{k} + K(z_{k}-Hx)}
    \boldsymbol{P}=\boldsymbol{(I - K\:H)\:\bar{P}}

    We must also define several variables. \boldsymbol{z_k} is our measurement – we get this directly from our input test function. This would be our noisy data for both x and y. \boldsymbol{\vec{x}} is initially set to all zeroes, except for \dot{x}. We assume that x will be linear and that the slope of x will be one, so we set \dot{x} to 1. \boldsymbol{P} is set to our initial certainty level or sigma squared of all of our variables. In this case we are unsure of almost everything except for \dot{x}, so we set everything to high values except for the part for \dot{x} which we set to 0 indicating that we are completely certain about this. Another way to think about this is “what is the expected error at the beginning” – the bigger the number, the more uncertain we are. \boldsymbol{H} is the measurement prediction function. It translates our state into the same format as our “measurement” – or x and y. We can get x and y from our state variable directly, so we don’t have to worry about nonlinearities, and we do not need a transfer function as they do in the book. \boldsymbol{R} is our matrix that tells us what we can expect the noise to be for our measurement – we use the standard deviation of the noise that we introduced in our function that we created to test the filter. All of these have been summarized below:
    \boldsymbol{\vec{x}_0} = \begin{bmatrix} 0 & 0 & 1 & 0 & 0 & 0 \end{bmatrix}
    \boldsymbol{H}  =  \begin{bmatrix} 1 & 0 & 0 & 0& 0& 0 \\  0 & 1 & 0 & 0 & 0 & 0 \\  \end{bmatrix}
    \boldsymbol{R} = \begin{bmatrix} \frac{1}{noise^2} & 0\\  0 & \frac{1}{noise^2} \end{bmatrix}
    \boldsymbol{P_0} =  \begin{bmatrix} 1000 & 0 & 0 &0 &0 &0\\  0 & 1000 & 0 & 0 & 0 & 0\\  0 & 0 & 0 & 0 & 0 &0 \\  0 & 0 & 0 & 1000 & 0 & 0 \\  0 & 0 & 0 & 0 & 1000 & 0 \\  0 & 0 & 0 & 0 & 0 & 1000 \\  \end{bmatrix}

    The only thing that we have left to define is our prediction function \boldsymbol{f}(\boldsymbol{\vec{x}},\delta t) and our Jacobian matrix \mathsf{F} Our prediction function looks like this:

    \boldsymbol{\vec{x}\hspace{1pt}'}  =  \boldsymbol{f}(\boldsymbol{\vec{x}},\delta t)
    \begin{bmatrix} x' \\ y' \\ \dot{x}' \\ a' \\ b' \\ c' \end{bmatrix}  = \begin{bmatrix} f_{1}(\boldsymbol{\vec{x}},\delta t) \\ f_{2}(\boldsymbol{\vec{x}},\delta t) \\ f_{3}(\boldsymbol{\vec{x}},\delta t) \\ f_{4}(\boldsymbol{\vec{x}},\delta t)\\ f_{5}(\boldsymbol{\vec{x}},\delta t) \\ f_{6}(\boldsymbol{\vec{x}},\delta t) \end{bmatrix}
    \: = \:  \begin{bmatrix} x + \dot{x}\, \delta t \\ a(x +\dot{x}\, \delta t )^2 + b(x + \dot{x}\, \delta t) +c \\ \dot{x} \\ a \\ b \\ c \end{bmatrix}
    \:=\:   \begin{bmatrix} x + \dot{x}\, \delta t \\ a\,x^2 + 2\, a\, x \dot{x}\, \delta t + a \, \dot{x}^2\, \delta t ^2 + b\, x + b\, \dot{x}\, \delta t + c\\ \dot{x} \\ a \\ b \\ c \end{bmatrix}

    From this we have to compute a Jacobian matrix. We compute a Jacobian by evaluating the derivative of the prediction function with respect to each of the variables of the state vector \boldsymbol{\vec{x}} . In other words our Jacobian \mathsf{F} is:
    \mathsf{F}  =  \begin{bmatrix} \frac{\delta f_{1}}{\delta x} & \frac{\delta f_{1}}{\delta y} & \frac{\delta f_{1}}{\delta \dot{x}} & \frac{\delta f_{1}}{\delta a}& \frac{\delta f_{1}}{\delta b} & \frac{\delta f_{1}}{\delta c} \\  \frac{\delta f_{2}}{\delta x} & \frac{\delta f_{2}}{\delta y} & \frac{\delta f_{2}}{\delta \dot{x}} & \frac{\delta f_{2}}{\delta a}& \frac{\delta f_{2}}{\delta b} & \frac{\delta f_{2}}{\delta c} \\  \frac{\delta f_{3}}{\delta x} & \frac{\delta f_{3}}{\delta y} & \frac{\delta f_{3}}{\delta \dot{x}} & \frac{\delta f_{3}}{\delta a}& \frac{\delta f_{3}}{\delta b} & \frac{\delta f_{3}}{\delta c} \\  \frac{\delta f_{4}}{\delta x} & \frac{\delta f_{4}}{\delta y} & \frac{\delta f_{4}}{\delta \dot{x}} & \frac{\delta f_{4}}{\delta a}& \frac{\delta f_{4}}{\delta b} & \frac{\delta f_{4}}{\delta c} \\  \frac{\delta f_{5}}{\delta x} & \frac{\delta f_{5}}{\delta y} & \frac{\delta f_{5}}{\delta \dot{x}} & \frac{\delta f_{5}}{\delta a}& \frac{\delta f_{5}}{\delta b} & \frac{\delta f_{5}}{\delta c} \\  \frac{\delta f_{6}}{\delta x} & \frac{\delta f_{6}}{\delta y} & \frac{\delta f_{6}}{\delta \dot{x}} & \frac{\delta f_{6}}{\delta a}& \frac{\delta f_{6}}{\delta b} & \frac{\delta f_{6}}{\delta c}  \end{bmatrix}

    Specifically, our Jacobian \mathsf{F} for our transfer function \boldsymbol{f}(\boldsymbol{\vec{x}},\delta t) is :
    \begin{bmatrix}  1 & 0 & \delta t & & & \\  2\,a\,x+2\,a\,\dot{x}\,\delta t+b & 0 & 2\,a\,x\,\delta t+2\,a\,\dot{x}\,\delta t^2+b\,\delta t & x^2+2\,x\,\dot{x}\,\delta t + \dot{x}^2\,\delta t^2 & x+\dot{x}\,\delta t & 1 \\  &0 & 1 & & & \\  &0 & & 1 & &\\  &0 & & & 1 &\\  &0 & & & & 1  \end{bmatrix}

    It is important to note that we must calculate our \mathsf{F} in each cycle through our filter.

    How did it work out? Actually, I was amazed! I set up my generating function to generate points around the function y=x^2-4x+20 and the output was \boldsymbol{\vec{x}} = \begin{bmatrix} 16.0304 & 212.0142 & 1.0000 & 1.0011 & -4.0852 & 20.2367\end{bmatrix} so it guessed a,b and c pretty close. Also the ending of x was 16, so 16^2-4 \cdot 16+20 = 212. Here is the output:

    ktest

    Here is the m code – for some reason I couldn’t upload it as a file

    parabola.m

    function [xpos ypos] = parabola
    
    
    noise = .5;
    dt = .01;
    tm = 16;
    a = 1;
    b = -4;
    c = 20;
    xpos = [];
    ypos = [];
    for t = 0 : dt :tm
    	   xpos = [xpos t+noise*randn ];
    	   ypos = [ypos a*t^2+b*t+c + noise*randn ];
              
    end
    len = length(xpos);
    startlen = 1;
    xpos = xpos(startlen:end);
    ypos = ypos(startlen:end);
    close all;
    plot(xpos,ypos);
    axis("equal")
    

    ktest_nonlinear2.m

    function [output] = ktest_nonlinear2
    [xdat ydat] = parabola;
    savedata = [xdat; ydat]';
    output = [];
    dt = .01;
    noise = .5;
    x = [0 0 1 0 0 0]' ;
    xdot = x;
    Q = [0 0 0 0;
         0 0 0 0;
         0 0 0 0;
         0 0 0 0];
    P = 1000* eye(6);
    P(3,3) = 0;
    
    R = [1/noise^2 0; 0 1/noise^2];
    H = [1 0  0 0 0 0;
         0 1  0 0 0 0];
    
    
    
    for element = 1:1:length(savedata)
        z = savedata(element,:)';
    
        x = f(x,dt) ;
        F = make_jacobian(x,dt);
        x= x';
        P = F*P*F';
        K = P*H'*(H*P*H' + R)^-1;
        
        x = x + K*(z-H*x);
    
    
        P = (eye(6) - (K * H)) * P;
        output = [output; x(1) x(2)];
        
    end
    x
    close all;
    hold all;
    plot(savedata(:,1),savedata(:,2),"b-");
    plot(output(:,1),output(:,2),"r:", "linewidth",2);
    legend('Measured', 'Estimated');
    title('Simulation of noisy X^2 data (y = x^2 + noise)');
    endfunction
    function state = f(vect,dt)
      x = vect(1);
      y = vect(2);
      xdot = vect(3);
      a = vect(4);
      b = vect(5);
      c = vect(6);
      statea = x+xdot*dt;
      stateb = a*x^2 + 2*a*x*xdot*dt + a * xdot^2 * dt^2 + b*x + b*xdot*dt + c;
      statec = xdot;
      stated = a;
      statee = b;
      statef = c;
      state = [statea stateb statec stated statee statef];
    	   
    endfunction
    
    
    function F = make_jacobian(vect,dt)
    x = vect(1);
    y = vect(2);
    xdot = vect(3);
    a = vect(4);
    b = vect(5);
    c = vect(6);
    F = [1 0 dt 0 0 0;
         2*a*x+2*a*xdot*dt+b 0 2*a*x*dt+2*a*xdot*dt^2+b*dt x^2+2*x*xdot*dt+xdot^2*dt^2 x+xdot*dt 1;
         0 0 1 0 0 0;
         0 0 0 1 0 0;
         0 0 0 0 1 0;
         0 0 0 0 0 1];
         
    
    
    endfunction
    

I got a BeagleBone black today, and since I like lisp, I tried to install CCL.

To my disappointment, it didn’t work. To make a long story short, I found that the /proc/cpuinfo was confusing the CPU check that CCL is doing. I’ve got a patch that I posted to the dev group, but thought I’d post it here too.

  1. Get CCL – I used subversion. Getting subversion on the BeagleBone is not too complicated, but it wasn’t too easy to find. I used instructions from here
  2. $ echo 'src/gz angstrom-base http://www.angstrom-distribution.org/feeds/unstable/ipk/glibc/armv7a/base'  > /etc/opkg/angstrom-base.conf
    $ opkg update
  3. svn co http://svn.clozure.com/publicsvn/openmcl/trunk/linuxarm/ccl
  4. Once this is checked out I have the following patch for ccl/lisp-kernel/pmcl-kernel.c
    --- pmcl-kernel.c (revision 15801)
    +++ pmcl-kernel.c (working copy)
    
    @@ -1604,7 +1604,7 @@
    break;
    }
    n = strlen(line);
    - if (strncmp(line,"Processor",sizeof("Processor")-1) == 0) {
    + if ((strncmp(line,"Processor",sizeof("Processor")-1) == 0) || (strncmp(line,"model name",sizeof("model name")-1)) == 0) {
    procline = malloc(n+1);
    strcpy(procline,line);
    procline[n]='';
  5. After patching the above file you go to ccl/lisp-kernel/linuxarm and then do a
    	make clean
    	make
  6. now you go back to the ccl directory and run ./armcl and it should work.  – good luck!

This is just in the hopes that it will help someone.

try using the :sharing :external option on open, this fixed my problem.

A longer description follows.

I was doing a simple test with CCL and slime, I was trying to open a file and call read multiple times. There could be any kind of lisp data in the file:

1 2 3 (1 2 3) :test :me

I tried to open and read from the file:


(setf testfile (open #p"~/fuzzy/test.data" ))
(read testfile)

this worked great the first time I tried it, but not the second time! I got an error.


Stream # is private to #
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [RETRY] Retry SLIME interactive evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT-BREAK] Reset this thread
 3: [ABORT] Kill this thread

Backtrace:
  0: (CCL::%PRIVATE-IOBLOCK-TYI #S(CCL::FILE-IOBLOCK :STREAM # :UNTYI-CHAR NIL :INBUF ...))
  1: (CCL::READ-CHAR-INTERNAL # NIL :EOF)
  2: (CCL::%NEXT-CHAR-AND-ATTR # #(1 1 1 1 1 1 ...))
  3: (CCL::%NEXT-NON-WHITESPACE-CHAR-AND-ATTR #)
  4: (CCL::%READ-FORM # 0 NIL)
  5: (CCL::READ-INTERNAL # T NIL NIL)
  6: (READ # T NIL NIL)
  7: (CCL::CALL-CHECK-REGS READ #)
  8: (CCL::CHEAP-EVAL (READ TESTFILE))
  9: ((:INTERNAL SWANK:INTERACTIVE-EVAL))
 --more--

after a bit of playing around, I realized that it worked OK from the repl, but not from a lisp file that I was working in.

I figured it had to do with accessing the file from different threads.
After a bit of digging through the source, I stumbled upon this fix:


(setf testfile (open #p"~/fuzzy/test.data" :sharing :external))

And I could access the file no matter where I was working from.

If you look in the CCL documentation

under heading 10.1.2. Additional keywords for OPEN and MAKE-SOCKET, you will see a bit of stuff about this option.

Hope this helps!

Since most of the traffic here is linking to the post I had on working with a lexer/yacc in lisp, I thought that I’d share a link to a post on a more ‘lispy’ way to approach the issue.

Paul says :

To a CL’er, infix notation is just unnecessary syntactic sugar. Many novice CL’ers try to create macros that understand infix notation early on in their careers, but as they become more familiar with the power of CL, they drop the syntatic sugar and use parse trees directly.

Which is true – the structure of lisp is exactly what you get when you run a bunch of infix code through a lex/yac or even an ANTLR combination.

The problem that I had when I was investigating the issue was that most people don’t think in prefix notation the way a lisper would! The prefix (also know as ‘abstract syntax tree’ or AST for short) notation is much easier to deal with from a programming perspective, which is why the first step for almost all compilers is to translate things down to AST format. This is why the above advice makes perfect sense if you are creating a DSL for yourself, but may not work if you are trying to create one for the rest of the world.

Just random thoughts . . .

This morning I accidentally deleted a bunch of stuff in a elephant store for lisp. I was using the standard berkeley DB. I was worried that everything I had done was totally gone.

Luckily you can do a db_recover to a specific date/time

db_recover -t

as a side note, I was trying to use db_printlog and it kept giving me an error about environment mismatch. After messing around with this for an obscene amount of time, I discovered that if you do much of anything with elephant, there are a bunch of __db.00x files out there. If you do a db_recover, (I would close your elephant store first, don’t know if it makes a difference) all of those files are gone and db_printlog (and a few other things) work fine. If you are sure that your executables are of the same flavor (i.e. 4.5 or 4.7 etc.) and you are having ‘version mismatch’ errors, give db_recover a try first.

Hope this helps!

What I did in the previous post is fine if we want to write a file right to disk, but what if we want to get the image data back from ImageMagick?

To do this we will need to use the #_MagickGetImageBlob call.

From the ImageMagick Documentation:

 MagickGetImageBlob

MagickGetImageBlob() implements direct to memory image formats.
 It returns the image as a blob and its length. 
Use MagickSetFormat() to set the format of the returned blob (GIF,
 JPEG, PNG, etc.).

Use MagickRelinquishMemory() to free the blob when you are done with it.

The format of the MagickGetImageBlob method is:

  unsigned char *MagickGetImageBlob(MagickWand *wand,size_t *length)
A description of each parameter follows:

wand

the magick wand.

length

the length of the blob.


We have already made plenty of calls where we use a pointer to MagickWand, but we have not yet done anything where we need to allocate memory to pass to our foreign function.

Luckily this isn’t too hard and this is a pretty trivial example.

First we set up the environment. I’m assuming we’ve done the stuff that I described in my first part of the tutorial, that is, set up the interface directory.

CL-USER> (ccl:use-interface-dir :imagemagick)
#

CL-USER> (ccl:use-interface-dir :libc)
#
; we load the libc interface so that we know what size size_t is

CL-USER>   (ccl:open-shared-library "libMagickWand.so")
#

CL-USER> (setf wand (#_NewMagickWand))
#<A Foreign Pointer#x1CB83F0 >

CL-USER> (#_MagickSetSize wand 100 100)
1

CL-USER> (with-cstrs ((image-format "png"))
	   (#_MagickSetFormat wand image-format))
;Compiler warnings :
;   In an anonymous lambda form: Undeclared free variable WAND
1

CL-USER> (with-cstrs ((red-image "xc:red"))
	   (#_MagickReadImage wand red-image))
;Compiler warnings :
;   In an anonymous lambda form: Undeclared free variable WAND
1

Now we have a wand that is a png, 100×100 and solid red. What we want to do next is allocate our memory, call our blob function, convert the return value to a string and free all of the memory.

To create the space for the length, we use the make-record macro (found in the FFI Documentation). Once we do this, we can call the Blob function.

The make-record macro retuns a pointer, and to be able to see what is in the memory at the pointer, we can use the ccl:pref function, and to free the pointer that we created, we use the ccl:free function. To translate c data to a string, we can use either ccl:%get-cstring if we know it is a normal C string (nul terminated) or, if we know the length, we can use ccl:%str-from-ptr (documented here


CL-USER> (setf length (ccl:make-record #>size_t))
#<A Foreign Pointer (:* (:UNSIGNED 64)) #x1D21080>

(ccl:pref length #>size_t)
0

CL-USER> (setf imagedata (#_MagickGetImageBlob wand length))
#<A Foreign Pointer #x7FCE070F1010 >

CL-USER> (ccl:pref length #>size_t)
477

CL-USER> (setf imagestr (ccl:%str-from-ptr imagedata (ccl:pref length #>size_t)))
"\211PNG^M
. . . bunch of stuff that I can't paste here"

CL-USER> (ccl:free length)
NIL

CL-USER> (#_MagickRelinquishMemory imagedata)
#<A Null Foreign Pointer>

CL-USER> (#_DestroyMagickWand wand)
#<A Null Foreign Pointer&gt



So, now we have gotten our string, we can push that out using hunchetoot (or whatever, just make sure you set the content-type to “image/png”), and we have cleaned up after ourselves.

Now that we have the ImageMagick bindings set up, we will try to do something ‘useful’ with what we have done.

I wanted to do something similar to what we find here that matches the C source code that we find here

so, we are combining the images:

Src Dest Mask

to end up with an image that looks like this:

Finished Image

Here is the lisp source:

; load imagemagick bindings
(use-interface-dir :imagemagick)

; load the shared library
(open-shared-library "libMagickWand.so")


; use with-cstrs to convert lisp strings to c style strings that
; we can pass to functions
(with-cstrs ((water "/home/kelly/blog/tile_water.jpg")
	     (aqua "/home/kelly/blog/tile_aqua.jpg")
	     (moon "/home/kelly/blog/moon_mask.gif")
	     (output "/home/kelly/blog/clip_out.jpg"))
  ; use #_ as a macro to reference functions
  (setf dest (#_NewMagickWand))
  (setf mask (#_NewMagickWand))
  (setf src (#_NewMagickWand))

  (#_MagickReadImage dest aqua)
  (#_MagickReadImage mask moon)
  ; use #$ as a macro to reference constants
  (#_MagickNegateImage mask #$MagickFalse)

  (#_MagickReadImage src water)
  ; use #> as a macro to reference structures
  ; and pref to dreference pointers
  (setf (ccl:pref dest #>MagickWand.images.mask) (ccl:pref mask #>MagickWand.images))

  (#_MagickCompositeImage dest src #$OverCompositeOp 0 0)
  ; null pointer with the %null-ptr macro
  (setf (ccl:pref dest #>MagickWand.images.mask) (ccl:%null-ptr))

  (#_MagickWriteImage dest output)

  (#_DestroyMagickWand src)
  (#_DestroyMagickWand dest)      
  (#_DestroyMagickWand mask))


As you can tell, we pretty much followed the C source code exactly, using the macros that Clozure provides (documentation)
If we wanted to do other things, we would use the ImageMagick API documentation here

The next post will show how to allocate memory and interface with a function that has a little more complicated call where we pass in a pointer of a data structure that we created in lisp and get back a pointer to memory.

I was working on creating on-the-fly images for a web application that I’m working on and off lately. I had made a method that called imagemagick using (run-program). This worked fine when I was running with one thread, but when I was using hunchentoot to serve it, threads were hanging up and acting flaky.

I went back to the drawing board and tried my hand at using FFI with Clozure. I was pleasantly suprised by what I found.

Clozure uses something called interface directories and comes with a few of these installed. You have to have the directories set up before you wan work with the FFI interface and that is what I’m going to cover in this post.

The first thing to know is that you need to have the ffigen4 tool installed. It wasn’t so easy to find the directions on installing this, but once you find them, it wasn’t too hard to get installed.

This installs a shell command h-to-ffi.sh that is used in the next step.

In order to use these you need to go to the ccl install directory (I just have it in my home directory) and go to the headers file that describes what version of CCL you are using. I’m using 64 bit CCL on linux so for me it would be ~/ccl/x86-headers64

you will see a few directories already installed there. Since I’m wanting to work with ImageMagick, I created an imagemagick directory and a C subdirectory per the instructions. The imagemagick directory should be all lowercase as CCL will later map a keyword to all lowercase to find the directory.

in the C directory, I created a new shell file populate.sh you must use this name exactly.

I put the following in the shell file:

#!/bin/sh
rm -rf ./home
h-to-ffi.sh -I/home/admin/ImageMagick-6.5.8-10 /home/admin/ImageMagick-6.5.8-10/magick/MagickCore.h
h-to-ffi.sh -I/home/admin/ImageMagick-6.5.8-10 /home/admin/ImageMagick-6.5.8-10/wand/MagickWand.h
h-to-ffi.sh -I/home/admin/ImageMagick-6.5.8-10 /home/admin/ImageMagick-6.5.8-10/wand/magick-wand-private.h

In most cases you can grab your header files out of your normal include directory (i.e. /usr/include) but because ImageMagick doesn’t install magick-wand-private.h, I needed to use the source directory for everything.

This is specific to ImageMagick, but I had to edit the magick-wand-private.h to include the line:

#include

at the top so that it could find all of the correct symbols.

once this is done, we can start up ccl and do the following (documented here):


?(require "PARSE-FFI")
PARSE-FFI

? (ccl::parse-standard-ffi-files :imagemagick)

note the double colons in the last statement, its not an exported function.

Assuming all went well you can now do the following


(use-interface-dir :imagemagick)
(open-shared-library "libMagickWand.so")
(setf wand (#_NewMagickWand))

and that will have to do until the next post.