Paul says :

To a CL’er, infix notation is just unnecessary syntactic sugar. Many novice CL’ers try to create macros that understand infix notation early on in their careers, but as they become more familiar with the power of CL, they drop the syntatic sugar and use parse trees directly.

Which is true – the structure of lisp is exactly what you get when you run a bunch of infix code through a lex/yac or even an ANTLR combination.

The problem that I had when I was investigating the issue was that most people don’t think in prefix notation the way a lisper would! The prefix (also know as ‘abstract syntax tree’ or AST for short) notation is much easier to deal with from a programming perspective, which is why the first step for almost all compilers is to translate things down to AST format. This is why the above advice makes perfect sense if you are creating a DSL for yourself, but may not work if you are trying to create one for the rest of the world.

Just random thoughts . . .

Luckily you can do a db_recover to a specific date/time

db_recover -t

as a side note, I was trying to use db_printlog and it kept giving me an error about environment mismatch. After messing around with this for an obscene amount of time, I discovered that if you do much of anything with elephant, there are a bunch of __db.00x files out there. If you do a db_recover, (I would close your elephant store first, don’t know if it makes a difference) all of those files are gone and db_printlog (and a few other things) work fine. If you are sure that your executables are of the same flavor (i.e. 4.5 or 4.7 etc.) and you are having ‘version mismatch’ errors, give db_recover a try first.

Hope this helps!

To do this we will need to use the #_MagickGetImageBlob call.

From the ImageMagick Documentation:

 MagickGetImageBlob

MagickGetImageBlob() implements direct to memory image formats.
 It returns the image as a blob and its length.
Use MagickSetFormat() to set the format of the returned blob (GIF,
 JPEG, PNG, etc.).

Use MagickRelinquishMemory() to free the blob when you are done with it.

The format of the MagickGetImageBlob method is:

  unsigned char *MagickGetImageBlob(MagickWand *wand,size_t *length)
A description of each parameter follows:

wand

the magick wand.

length

the length of the blob.

We have already made plenty of calls where we use a pointer to MagickWand, but we have not yet done anything where we need to allocate memory to pass to our foreign function.

Luckily this isn’t too hard and this is a pretty trivial example.

First we set up the environment. I’m assuming we’ve done the stuff that I described in my first part of the tutorial, that is, set up the interface directory.

CL-USER> (ccl:use-interface-dir :imagemagick)
#

CL-USER> (ccl:use-interface-dir :libc)
#
; we load the libc interface so that we know what size size_t is

CL-USER>   (ccl:open-shared-library "libMagickWand.so")
#

CL-USER> (setf wand (#_NewMagickWand))
#<A Foreign Pointer#x1CB83F0 >

CL-USER> (#_MagickSetSize wand 100 100)
1

CL-USER> (with-cstrs ((image-format "png"))
	   (#_MagickSetFormat wand image-format))
;Compiler warnings :
;   In an anonymous lambda form: Undeclared free variable WAND
1

CL-USER> (with-cstrs ((red-image "xc:red"))
	   (#_MagickReadImage wand red-image))
;Compiler warnings :
;   In an anonymous lambda form: Undeclared free variable WAND
1

Now we have a wand that is a png, 100×100 and solid red. What we want to do next is allocate our memory, call our blob function, convert the return value to a string and free all of the memory.

To create the space for the length, we use the make-record macro (found in the FFI Documentation). Once we do this, we can call the Blob function.

The make-record macro retuns a pointer, and to be able to see what is in the memory at the pointer, we can use the ccl:pref function, and to free the pointer that we created, we use the ccl:free function. To translate c data to a string, we can use either ccl:%get-cstring if we know it is a normal C string (nul terminated) or, if we know the length, we can use ccl:%str-from-ptr (documented here

CL-USER> (setf length (ccl:make-record #>size_t))
#<A Foreign Pointer (:* (:UNSIGNED 64)) #x1D21080>

(ccl:pref length #>size_t)
0

CL-USER> (setf imagedata (#_MagickGetImageBlob wand length))
#<A Foreign Pointer #x7FCE070F1010 >

CL-USER> (ccl:pref length #>size_t)
477

CL-USER> (setf imagestr (ccl:%str-from-ptr imagedata (ccl:pref length #>size_t)))
"\211PNG^M
. . . bunch of stuff that I can't paste here"

CL-USER> (ccl:free length)
NIL

CL-USER> (#_MagickRelinquishMemory imagedata)
#<A Null Foreign Pointer>

CL-USER> (#_DestroyMagickWand wand)
#<A Null Foreign Pointer&gt

So, now we have gotten our string, we can push that out using hunchetoot (or whatever, just make sure you set the content-type to “image/png”), and we have cleaned up after ourselves.

I wanted to do something similar to what we find here that matches the C source code that we find here

so, we are combining the images:

to end up with an image that looks like this:

Here is the lisp source:

; load imagemagick bindings
(use-interface-dir :imagemagick)

; load the shared library
(open-shared-library "libMagickWand.so")

; use with-cstrs to convert lisp strings to c style strings that
; we can pass to functions
(with-cstrs ((water "/home/kelly/blog/tile_water.jpg")
	     (aqua "/home/kelly/blog/tile_aqua.jpg")
	     (moon "/home/kelly/blog/moon_mask.gif")
	     (output "/home/kelly/blog/clip_out.jpg"))
  ; use #_ as a macro to reference functions
  (setf dest (#_NewMagickWand))
  (setf mask (#_NewMagickWand))
  (setf src (#_NewMagickWand))

  (#_MagickReadImage dest aqua)
  (#_MagickReadImage mask moon)
  ; use #$ as a macro to reference constants
  (#_MagickNegateImage mask #$MagickFalse)

  (#_MagickReadImage src water)
  ; use #> as a macro to reference structures
  ; and pref to dreference pointers
  (setf (ccl:pref dest #>MagickWand.images.mask) (ccl:pref mask #>MagickWand.images))

  (#_MagickCompositeImage dest src #$OverCompositeOp 0 0)
  ; null pointer with the %null-ptr macro
  (setf (ccl:pref dest #>MagickWand.images.mask) (ccl:%null-ptr))

  (#_MagickWriteImage dest output)

  (#_DestroyMagickWand src)
  (#_DestroyMagickWand dest)
  (#_DestroyMagickWand mask))

As you can tell, we pretty much followed the C source code exactly, using the macros that Clozure provides (documentation)
If we wanted to do other things, we would use the ImageMagick API documentation here

The next post will show how to allocate memory and interface with a function that has a little more complicated call where we pass in a pointer of a data structure that we created in lisp and get back a pointer to memory.

I went back to the drawing board and tried my hand at using FFI with Clozure. I was pleasantly suprised by what I found.

Clozure uses something called interface directories and comes with a few of these installed. You have to have the directories set up before you wan work with the FFI interface and that is what I’m going to cover in this post.

The first thing to know is that you need to have the ffigen4 tool installed. It wasn’t so easy to find the directions on installing this, but once you find them, it wasn’t too hard to get installed.

This installs a shell command h-to-ffi.sh that is used in the next step.

In order to use these you need to go to the ccl install directory (I just have it in my home directory) and go to the headers file that describes what version of CCL you are using. I’m using 64 bit CCL on linux so for me it would be ~/ccl/x86-headers64

you will see a few directories already installed there. Since I’m wanting to work with ImageMagick, I created an imagemagick directory and a C subdirectory per the instructions. The imagemagick directory should be all lowercase as CCL will later map a keyword to all lowercase to find the directory.

in the C directory, I created a new shell file populate.sh you must use this name exactly.

I put the following in the shell file:

#!/bin/sh
rm -rf ./home
h-to-ffi.sh -I/home/admin/ImageMagick-6.5.8-10 /home/admin/ImageMagick-6.5.8-10/magick/MagickCore.h
h-to-ffi.sh -I/home/admin/ImageMagick-6.5.8-10 /home/admin/ImageMagick-6.5.8-10/wand/MagickWand.h
h-to-ffi.sh -I/home/admin/ImageMagick-6.5.8-10 /home/admin/ImageMagick-6.5.8-10/wand/magick-wand-private.h

In most cases you can grab your header files out of your normal include directory (i.e. /usr/include) but because ImageMagick doesn’t install magick-wand-private.h, I needed to use the source directory for everything.

This is specific to ImageMagick, but I had to edit the magick-wand-private.h to include the line:

#include

at the top so that it could find all of the correct symbols.

once this is done, we can start up ccl and do the following (documented here):


?(require "PARSE-FFI")
PARSE-FFI

? (ccl::parse-standard-ffi-files :imagemagick)


note the double colons in the last statement, its not an exported function.

Assuming all went well you can now do the following


(use-interface-dir :imagemagick)
(open-shared-library "libMagickWand.so")
(setf wand (#_NewMagickWand))


and that will have to do until the next post.

no matter how I adjusted the paths, it would not load the library with uffi.

After messing around with it for most of the morning, I finally discovered that there was something funky with the symbolic links in the /usr/lib/ area.

lrwxrwxrwx 1 root root     15 2009-12-30 08:09 libsqlite3.so -> libqslite3.so.0

lrwxrwxrwx 1 root root     19 2009-11-02 23:03 libsqlite3.so.0 -> libsqlite3.so.0.8.6

-rw-r–r– 1 root root 556648 2009-09-19 16:35 libsqlite3.so.0.8.6

when I tried to ldd /usr/lib/libsqlite3.so it said that there was no file /usr/lib/libsqlite3.so

Redoing the link like so:

sudo ln -sf libsqlite3.so.0.8.6 libsqlite3.so

fixed the problem and allowed me to (require ‘clsql-sqlite3) with no problems

HTH

(defn plusn [x]
(fn [y] (+ x y)))

(def plus5 (plusn 5))
(plus5 3)

>> 8

Yep, it’s got closures

Of course, knowing that Lisp is well suited to working with compilers,
I wanted to play with some of this stuff in lisp.

I loaded cl-yacc and messed with their list lexer that they had in the
examples. This led me to wanting to get a more complex lexer that
would work well with a file or string.

Of course I could just use the lisp read which is in essence a lexer,
it doesn’t handle things like “parse this line;” where you want the
semicolon to be a token in it’s own right. Lisp read will happily pull
in the last symbol ‘line;’.

So . . I used ASDF to load up regex and clawk from cliki, and
downloaded lexer from wherever Google led me. This is where my
problems began – and I’m hoping that I can avoid you some pain.

The lexer that I downloaded referenced a function
macroexpand-regex-expr which was not available, and when I tried to
figure out what it might be referencing, I still had problems. Perhaps
I would have gotten to the end of it, but then again, maybe not.

The right thing to do (in this case) was to use the lispbuilder code
that is at sourceforge. Because
they weren’t immediately installable with asdf-install I had to
extract them and re tgz them. I added these to cliki so
that we can just asdf-install them. Just for grins, here is some of
the code that I’m playing with.

 

(defparameter test-lexer)

(lispbuilder-lexer:deflexer test-lexer
    ("[0-9]+([.][0-9]+([Ee][0-9]+)?)"
      (return (values 'flt (read-from-string lispbuilder-lexer:%0))))
    ("[0-9]+"
      (return (values 'int (parse-integer lispbuilder-lexer:%0))))
    ("[:alpha:][:alnum:]*"
      (return (values 'id (read-from-string lispbuilder-lexer:%0))))
    ("(-|[+*/])"
      (return (values (read-from-string lispbuilder-lexer:%0) (read-from-string lispbuilder-lexer:%0))))
    ("[(]"
      (return (values '|(| '|(|)))
    ("[)]"
      (return (values '|)| '|)|)))
    ("[:space:]+") )

(defvar *expression-parser*)

(yacc:define-parser *expression-parser*
  (:start-symbol expression)
  (:terminals (int id + - * / |(| |)|))
  (:precedence ((:left * /) (:left + -)))

  (expression
   (expression + expression #'i2p)
   (expression - expression #'i2p)
   (expression * expression #'i2p)
   (expression / expression #'i2p)
   term)

  (term
   id
   int
   (- term)
   (|(| expression |)| #'k-2-3)))

(setq *test* (yacc:parse-with-lexer (test-lexer "x * - (2 + 3) * y") *expression-parser*))

(print *test*)

 (* (* X (- (+ 2 3))) Y)

This book is an introduction to interpreters and compilers. It is intended to
be down a level from the dragon book. I found the book accessible, an
easy read for the most part. (I read the book mostly over the weekend,
but I did do some skimming) If anything, it connected the
dots just a little too much, but the book stays true to it’s purpose.

Kaplan starts with a survey of little languages, and discusses design
of languages. That discussion was one of the reasons why I bought the
book and I was a bit disappointed with that aspect because it
covered just a few pages (37-39). Other parts of the tour were quite
adequate and you come away with a firm grasp of what a lexer and a
parser do as well as a basic understanding of lex & yacc and how all
of this stuff is brought together to make an interpreter or a
compiler. I particularly enjoyed the section on computation engines &
finite state automata (82 ff)

The author is careful to point out that the standard book on the topic
is the dragon book, and points you to it in several instances.

Most of the book revolves around a language that the author made up
for a specific purpose and takes if cradle to grave. He develops a
grammar, lexer, parser (both from scratch and using lex & yacc) and
shows in very fine detail (too much at times) how each fits into the
overall picture. All source code is given in c, often with a walk
through of each procedure and data structure.

How does this relate to lisp? Well he does have an interesting chapter
in the back showing how do manage the same task (i.e. a little
language) in both prolog and lisp. He doesn’t start with the same
input as he uses for his grammar and example in the rest of the book
but shows how to modify the grammar slightly to allow you to write
your little language. I would certainly like to revisit both of those
sections, but they are just a tiny part of the book. I was hoping to
find some references to fuller works on the topic with these languages
in mind. (for lisp he pointed to CLTL2 and SICP)

Also, it has been a very long time since I have programmed in c. It
was striking how verbose the source code really was for the
project. After he presents the prolog code for one of the statements
he comments: “This may seem like a lot of code, but it translates and
executes …”.

In the last chapter, it seems that he is making the case for a richer
environment for making little languages and that prolog, lisp and
forth were all better platforms for creating them, even
with the helper programs lex and yacc. However, as I read along and
noticed several places where things would have been much easier in
lisp it may have been mere projection on my part.

I am glad that I read the book, but I think that I probably should
have saved my book buying $$ for the dragon book and tried to borrow
this one. On the other hand, I actually got through this book and I
found it useful for thinking about the upcoming design of a little
language, and I don’t know if either could be said about the dragon
book.

I thought I would solve the problem by making a macro so it would look
something like:

(cond (macro parm1 parm2 parm3)
      (macro parm1 parm2 parm3))

Any lisp veteran is probably chuckling to themselves thinking that I
probably have also tried to do the functional equivalent of something like:

(setq x '(pred sexpr))

(cond x)

As I learned quite quickly, you just can’t do this because of the
macro expansion of cond.

>(print (macroexpand-1 '(cond (macro param1 param2)
		      (macro param1 param2))))

(IF MACRO (PROGN PARAM1 PARAM2) (COND (MACRO PARAM1 PARAM2)))

The problem in both of these scenarios is that the cond macro gets
expanded before any of the other macros, essentially goofing up our
original plan.

To solve the problem I just changed it around so I had a macro that
took the list of parameters that I wanted and created the entire cond
statement for me.

(cond-macro (param1 param2)
	    (param1 param2))

Which did exactly what I wanted.