changelog shortlog graph tags branches changeset files file revisions raw help

Mercurial > core / annotate lisp/ffi/rocksdb/pkg.lisp

changeset 273: 6d56c4950fa2
parent: 4a1a21ff46ee
child: 00d1c8afcdbb
author: Richard Westhaver <ellis@rwest.io>
date: Wed, 10 Apr 2024 22:09:23 -0400
permissions: -rw-r--r--
description: rocksdb iter,metadata,and more
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
1
 ;;; rocksdb.lisp --- low-level bindings to the RocksDB C API
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
2
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
3
 ;; for the high-level interface, see rdb.lisp.
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
4
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
5
 ;;; Commentary:
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
6
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
7
 ;; if ur on archlinux and installed rocksdb via AUR you may receive an error from
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
8
 ;; jemalloc: cannot allocate memory in static TLS block:
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
9
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
10
 ;; https://github.com/veer66/cl-rocksdb/issues/1
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
11
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
12
 ;; for best results, you should compile rocksdb from source - use j0ni's snippet as a
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
13
 ;; starting point.
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
14
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
15
 ;; make shared_lib DISABLE_JEMALLOC=1 && 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
16
 ;; sudo cp librocksdb.so.* /usr/local/lib/ && 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
17
 ;; sudo cp -rf include/* /usr/local/include/
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
18
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
19
 ;; https://github.com/facebook/rocksdb/blob/main/Makefile
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
20
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
21
 ;; check /usr/local/include/rocksdb/c.h for the C API header, the source is under
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
22
 ;; db/c.cc
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
23
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
24
 ;; here are some important notes to keepin mind (from the API header):
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
25
 #|
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
26
 C bindings for rocksdb.  May be useful as a stable ABI that can be
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
27
 used by programs that keep rocksdb in a shared library, or for
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
28
 a JNI api.
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
29
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
30
 Does not support:
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
31
 . getters for the option types
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
32
 . custom comparators that implement key shortening
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
33
 . capturing post-write-snapshot
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
34
 . custom iter, db, env, cache implementations using just the C bindings
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
35
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
36
 Some conventions:
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
37
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
38
 (1) We expose just opaque struct pointers and functions to clients.
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
39
 This allows us to change internal representations without having to
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
40
 recompile clients.
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
41
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
42
 (2) For simplicity, there is no equivalent to the Slice type.  Instead,
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
43
 the caller has to pass the pointer and length as separate
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
44
 arguments.
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
45
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
46
 (3) Errors are represented by a null-terminated c string.  NULL
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
47
 means no error.  All operations that can raise an error are passed
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
48
 a "char** errptr" as the last argument.  One of the following must
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
49
 be true on entry:
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
50
 *errptr == NULL
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
51
 *errptr points to a malloc()ed null-terminated error message
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
52
 On success, a leveldb routine leaves *errptr unchanged.
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
53
 On failure, leveldb frees the old value of *errptr and
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
54
 set *errptr to a malloc()ed error message.
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
55
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
56
 (4) Bools have the type unsigned char (0 == false; rest == true)
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
57
 
47
01f7dc4d7a8e rocksdb macros
ellis <ellis@rwest.io>
parents: 45
diff changeset
58
 (5) All of the pointer arguments must be non-NULL.|#
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
59
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
60
 ;;; Code:
85
1088a94a6673 rocksdb opts
ellis <ellis@rwest.io>
parents: 84
diff changeset
61
 (defpackage :rocksdb
96
301fd45bbe73 big refactor of lisp code
ellis <ellis@rwest.io>
parents: 94
diff changeset
62
   (:use :cl :std :sb-alien)
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
63
   (:export
113
afcb1f02bb66 removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents: 112
diff changeset
64
    ;; vars
94
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 90
diff changeset
65
    :*rocksdb-options*
110
cae8da4b1415 rocksdb macrofication, fixes for RDB (C Strings will be the death of me), removed PWN - quicklisp package conflict, smh.
ellis <ellis@rwest.io>
parents: 101
diff changeset
66
    :*rocksdb-compaction-levels*
cae8da4b1415 rocksdb macrofication, fixes for RDB (C Strings will be the death of me), removed PWN - quicklisp package conflict, smh.
ellis <ellis@rwest.io>
parents: 101
diff changeset
67
    :*rocksdb-compression-backends*
269
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 155
diff changeset
68
    :rocksdb-compression-backend
110
cae8da4b1415 rocksdb macrofication, fixes for RDB (C Strings will be the death of me), removed PWN - quicklisp package conflict, smh.
ellis <ellis@rwest.io>
parents: 101
diff changeset
69
    :*rocksdb-perf-metrics*
269
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 155
diff changeset
70
    :*rocksdb-perf-levels*
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 155
diff changeset
71
    :*rocksdb-statistics-levels*
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 271
diff changeset
72
    :rocksdb-statistics-level
271
4a1a21ff46ee metadata,props,merge-ops
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
73
    :rocksdb-perf-level
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 271
diff changeset
74
    :*rocksdb-column-family-metadata*
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 271
diff changeset
75
    :*rocksdb-level-metadata*
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 271
diff changeset
76
    :*rocksdb-sst-file-metadata*
271
4a1a21ff46ee metadata,props,merge-ops
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
77
    :*rocksdb-properties*
4a1a21ff46ee metadata,props,merge-ops
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
78
    :rocksdb-num-files-at-level
4a1a21ff46ee metadata,props,merge-ops
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
79
    :rocksdb-compression-ratio-at-level
4a1a21ff46ee metadata,props,merge-ops
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
80
    :rocksdb-aggregated-table-properties-at-level))
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
81
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
82
 (in-package :rocksdb)
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
83
 
155
b4bb3c52bc18 moved external ffi libs
ellis <ellis@rwest.io>
parents: 149
diff changeset
84
 (define-alien-loader "rocksdb" t)