lmdb-ht

Documentation

The lmdb-ht library provides bindings for the Lightning Memory-Mapped Database Manager (LMDB) database management library, with optional encryption of keys and values.

Requirements

The LMDB C library and its header file lmdb.h (on Debian and Ubuntu, the liblmdb-dev package).
rabbit and srfi-69.

If LMDB is installed in a non-standard location, set the environment variables LMDB_CFLAGS, LMDB_LFLAGS and LMDB_LIBS before running chicken-install. All three must be set.

Keys and values

All keys and values are blobs. Use string->blob or a serialization library to convert other Scheme objects.

Encryption

If db-open is given an encryption key, every key and value is encrypted with the rabbit stream cipher before it is stored, and decrypted when it is read.

The encryption key is a blob of at least 16 bytes. Only the first 16 bytes are used. With rabbit 3.0, the key must also be valid UTF-8, for example printable ASCII. A key made of arbitrary random bytes makes db-open signal an error.

The encryption is done in place: db-set!, db-ref and db-delete overwrite the key and value blobs passed to them. Pass fresh blobs if you need the original contents afterwards.

Transactions

All operations other than db-open, db-close, db-delete-database and db-max-key-size must happen inside a transaction started with db-begin and finished with db-end or db-abort.

Errors

LMDB errors are signaled as composite conditions of kinds exn, lmdb and a kind that names the LMDB error code. The error code kinds are:

mdb-keyexist, mdb-notfound, mdb-page-notfound, mdb-corrupted, mdb-panic, mdb-version-mismatch, mdb-invalid, mdb-map-full, mdb-dbs-full, mdb-readers-full, mdb-tls-full, mdb-txn-full, mdb-cursor-full, mdb-page-full, mdb-map-resized, mdb-incompatible, mdb-bad-rslot, mdb-bad-txn, mdb-bad-valsize, mdb-bad-dbi, and unknown for any other code.

For example, to catch a missing key:

(condition-case (db-ref mm (string->blob "no-such-key"))
  ((exn lmdb mdb-notfound) 'missing))

Library procedures

Databases

(db-open dbname [key: enckey] [maxdbs: n] [mapsize: size])procedure

Opens or creates the LMDB database in directory dbname, creating the directory if needed, and returns a database handle.

enckey: optional encryption key blob (see Encryption). The default is no encryption.
maxdbs: the maximum number of named databases in the environment. It must be greater than 0 to use the dbname: argument of db-begin.
mapsize: the size of the memory map in bytes, which is also the maximum size of the database. When it is 0, the LMDB default is used.

db-close dbprocedure: Closes the database handle and frees its encryption context.

db-delete-database dbnameprocedure: Deletes the LMDB database in directory dbname, if it exists. The database must be closed.

db-max-key-size dbprocedure: Returns the maximum key size, in bytes, that the database accepts.

Transactions

(db-begin db [dbname: dbname] [flags: flags])procedure: Begins a transaction. dbname selects a named database (see maxdbs: in db-open); when it is #f, the default, the main database is used. flags is a bit mask returned by db-flags; the default is 0.

db-end dbprocedure: Commits and ends the current transaction.

db-abort dbprocedure

Aborts the current transaction and discards its changes.

If db-set! signals an error, the transaction stays open. End it with db-end or db-abort.

db-flags flag-name #!optional seedprocedure

Returns the bit mask for an LMDB flag, combined with the optional seed mask. To combine several flags, pass the result of one call as the seed of the next. Returns #f for an unknown flag name. The flag names are:

#:fixed-map: mmap at a fixed address
#:no-subdirectory: no environment directory
#:read-only: read only
#:write-map: use a writable mmap
#:no-meta-sync: do not fsync the metapage after commit
#:no-sync: do not fsync after commit
#:map-async: use asynchronous sync when #:write-map is used
#:no-lock: do not do locking
#:no-read-ahead: do not do readahead

Reading and writing

db-set! db key valueprocedure: Stores value under key, replacing any existing value.

db-ref db keyprocedure: Returns the value stored under key. Signals an (exn lmdb mdb-notfound) condition if the key is not present.

db-delete db keyprocedure: Removes key and its value. Signals an (exn lmdb mdb-notfound) condition if the key is not present.

db-count dbprocedure: Returns the number of key-value pairs in the database.

Iteration

These procedures signal an (exn lmdb mdb-notfound) condition when the database is empty.

LMDB keeps pairs sorted by the stored key bytes. In an unencrypted database this is ascending key order. In an encrypted database the stored bytes are encrypted, so the order does not follow the plain keys.

db-keys dbprocedure: Returns a list of all keys, in the reverse of the stored order.

db-values dbprocedure: Returns a list of all values, in the reverse of the stored order.

db-fold f init dbprocedure: Folds over the key-value pairs in the stored order. f is called as (f key value acc) and returns the new accumulator; init is the first accumulator value.

db-for-each f dbprocedure: Calls (f key value) for each key-value pair in the stored order.

Hash tables

(hash-table->db table dbname [key: enckey])procedure: Deletes the database in directory dbname, then creates it again and stores the contents of the SRFI-69 hash table table. The hash table keys must be symbols, and the values must be blobs. Returns #t.

(db->hash-table dbname [key: enckey])procedure: Loads the database in directory dbname into a new SRFI-69 hash table and returns it. The hash table keys and values are blobs. Returns #f if the directory does not exist.

Cursor access

These procedures read the database one key-value pair at a time. They must be called inside a transaction.

db-index-first dbprocedure
db-index-next dbprocedure: Moves the cursor to the first or next key-value pair, in the stored order. Returns 0 on success, or an LMDB error code, such as the code for mdb-notfound after the last pair.

db-key-len dbprocedure
db-value-len dbprocedure: Returns the length in bytes of the key or value at the cursor.

db-key db bufprocedure

db-value db bufprocedure

Copies the key or value at the cursor into the blob buf, decrypts it if the database is encrypted, and returns buf. Signals an error if buf is shorter than db-key-len or db-value-len.

(db-begin mm)
(let loop ((rc (db-index-first mm)))
  (when (= rc 0)
    (print (db-key mm (make-blob (db-key-len mm))))
    (loop (db-index-next mm))))
(db-end mm)

Debugging

debuglevel #!optional nparameter: Sets the level of debug messages printed to standard output. The default is 0, which prints nothing.

Examples

;; Encrypted key-value creation and lookup

(import scheme (chicken base) (chicken blob) (chicken pathname)
        (chicken string) lmdb-ht)

(let* ((fname (make-pathname "." "mydb.mdb"))
       (keys (list "k1" 'k2 '(k3)))
       (values (list 'one 2 "three"))
       (cryptokey (string->blob "0123456789abcdef"))
       (mm (db-open fname key: cryptokey)))
  (db-begin mm)
  (for-each (lambda (k v)
              (db-set! mm (string->blob (->string k)) (string->blob (->string v))))
            keys values)
  (db-end mm)
  (db-begin mm)
  (let ((res (let loop ((ks keys) (vs values))
               (or (null? ks)
                   (and (equal? (string->blob (->string (car vs)))
                                (db-ref mm (string->blob (->string (car ks)))))
                        (loop (cdr ks) (cdr vs)))))))
    (db-end mm)
    (db-close mm)
    res))

;; Read-only transaction

(let ((mm (db-open (make-pathname "." "mydb.mdb"))))
  (db-begin mm)
  (db-set! mm (string->blob "foo") (string->blob "one"))
  (db-end mm)
  (db-begin mm flags: (db-flags #:read-only))
  (print (blob->string (db-ref mm (string->blob "foo"))))
  (db-end mm)
  (db-close mm))

3.6: Cursor procedures take the database handle; db->hash-table decrypts all entries; a failed db-set! no longer ends the transaction
3.5: Ported to CHICKEN 6
3.4: Added db-flags and read-only transactions [thanks to alice maz]
3.3: LMDB errors are raised from Scheme instead of C [thanks to alice maz]
3.2: Added db-delete [thanks to alice maz and Caolan McMahon]
3.0: Ported to CHICKEN 5
2.3: Using (exn lmdb ...) style composite conditions to signal lmdb exceptions [thanks to Caolan McMahon].
2.0: Added lmdb-begin/end procedures for explicit transaction control.
1.0: Initial release

License

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

  Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

  Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

  Neither the name of the author nor the names of its contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.

chickadee » lmdb-ht

Identifier search

TOC »

Documentation

Requirements

Keys and values

Encryption

Transactions

Errors

Library procedures

Databases

Transactions

Reading and writing

Iteration

Hash tables

Cursor access

Debugging

Examples

About this egg

Author

Repository

Version history

License

Contents »