changeset 599: | fea71448569b |
parent: | c7f9bfc9570f |
child: | 6fc04c4d465c |
author: | Richard Westhaver <ellis@rwest.io> |
date: | Fri, 16 Aug 2024 18:06:48 -0400 |
permissions: | -rw-r--r-- |
description: | more rocksdb gruntwork |
271 | 1 | ;;; rocksdb/merge.lisp --- RocksDB Merge Operators |
2 | ||
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
3 | ;; RocksDB Lisp Merge Operator API |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
4 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
5 | ;;; Commentary: |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
6 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
7 | ;; When to use built-in ROCKSDB-MERGE: |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
8 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
9 | ;; - You have data that needs to be incrementally updated. |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
10 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
11 | ;; - You would usually need to read the data before knowing what the new value would be. |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
12 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
13 | ;; Oterwise as far as the FFI is concerned - which doesn't support |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
14 | ;; AssociateMerge, you should use the Generic Merge API. |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
15 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
16 | ;; When to use Associative Merge (unavailable in C/LISP API): |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
17 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
18 | ;; - merge operands are formatted the same as Put values AND |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
19 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
20 | ;; - it is okay to combine multiple operands into one |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
21 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
22 | ;; When to use Generic Merge (this API): |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
23 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
24 | ;; - you are unable to use Associate Merge |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
25 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
26 | ;; - it is possible to combine multiple operands |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
27 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
28 | ;;; Refs: |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
29 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
30 | ;; impl: https://github.com/facebook/rocksdb/wiki/Merge-Operator-Implementation |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
31 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
32 | ;; wiki: https://github.com/facebook/rocksdb/wiki/merge-operator |
271 | 33 | |
34 | ;;; Code: |
|
35 | (in-package :rocksdb) |
|
36 | ||
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
37 | #| |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
38 | Gives the client a way to express the read -> modify -> write semantics |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
39 | key: (IN) The key that's associated with this merge operation. |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
40 | existing: (IN) null indicates that the key does not exist before this op |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
41 | operand_list:(IN) the sequence of merge operations to apply, front() first. |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
42 | new_value: (OUT) Client is responsible for filling the merge result here |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
43 | logger: (IN) Client could use this to log errors during merge. |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
44 | |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
45 | Return true on success. Return false failure / error / corruption. |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
46 | |# |
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
47 | ;; FullMerge() is used when a Put/Delete is the *existing_value (or null) |
271 | 48 | (define-alien-type rocksdb-full-merge-function |
49 | (function (* t) |
|
50 | (array unsigned-char) |
|
51 | size-t |
|
52 | (array (array unsigned-char)) |
|
53 | (array size-t) |
|
54 | int |
|
55 | (array unsigned-char) |
|
56 | (* size-t))) |
|
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
57 | |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
58 | #| |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
59 | This function performs merge(left_op, right_op) |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
60 | when both the operands are themselves merge operation types. |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
61 | Save the result in *new_value and return true. If it is impossible |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
62 | or infeasible to combine the two operations, return false instead. |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
63 | |# |
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
64 | ;; PartialMerge() is used to combine two-merge operands (if possible) |
271 | 65 | (define-alien-type rocksdb-partial-merge-function |
66 | (function (* t) |
|
67 | (array unsigned-char) |
|
68 | size-t |
|
69 | (array (array unsigned-char)) |
|
70 | (array size-t) |
|
71 | int |
|
72 | (array unsigned-char) |
|
73 | (* size-t))) |
|
74 | ||
75 | (define-alien-type rocksdb-delete-value-function |
|
76 | (function (* t) |
|
77 | (array unsigned-char) |
|
78 | size-t)) |
|
79 | ||
80 | (define-alien-type rocksdb-destructor-function |
|
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
81 | (function void (* t))) |
271 | 82 | |
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
83 | #| |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
84 | The name of the MergeOperator. Used to check for MergeOperator |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
85 | mismatches (i.e., a DB created with one MergeOperator is |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
86 | accessed using a different MergeOperator) |
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
87 | |# |
271 | 88 | (define-alien-type rocksdb-name-function |
89 | (function c-string)) |
|
90 | ||
91 | (deftype rocksdb-merge-operands () '(array (octet-vector))) |
|
92 | ||
93 | ;; (sb-alien::define-alien-callable mangle int () 0) |
|
94 | ||
95 | (define-alien-routine rocksdb-mergeoperator-create (* rocksdb-mergeoperator) |
|
96 | (state (* t)) |
|
97 | (destructor (* rocksdb-destructor-function)) |
|
98 | (full-merge (* rocksdb-full-merge-function)) |
|
99 | (partial-merge (* rocksdb-partial-merge-function)) |
|
100 | (delete-value (* rocksdb-delete-value-function)) |
|
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
101 | (name (* rocksdb-name-function))) |
271 | 102 | |
103 | #| [[file:~/dev/comp/core/c/rocksdb.h::/* Merge Operator */]] |# |
|
104 | ||
105 | (define-alien-routine rocksdb-mergeoperator-destroy void (self (* rocksdb-mergeoperator))) |
|
106 | ||
107 | ;; TODO 2023-12-11: |
|
108 | (deftype rocksdb-mergeoperator-function () |
|
109 | '(function (octet-vector (or octet-vector null) &rest t) (or null octet-vector))) |
|
110 | ||
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
111 | (define-alien-callable rocksdb-delete-value (* t) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
112 | ((val (array unsigned-char)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
113 | (vlen size-t)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
114 | (declare (ignore val vlen)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
115 | nil) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
116 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
117 | (define-alien-callable rocksdb-destructor void ((self (* t))) |
598
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
118 | (free-alien self) |
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
119 | (values)) |
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
120 | |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
121 | (define-alien-callable rocksdb-name c-string () (make-alien-string (symbol-name (gensym "rocksdb:")))) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
122 | |
599 | 123 | (define-alien-callable rocksdb-concat-merge-name c-string () (make-alien-string "concat-merge")) |
124 | ||
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
125 | (define-alien-callable rocksdb-concat-full-merge boolean |
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
126 | ((key (array unsigned-char)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
127 | (klen size-t) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
128 | (existing-val (array unsigned-char)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
129 | (existing-vlen size-t) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
130 | (ops (array (array unsigned-char))) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
131 | (ops-length (* size-t)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
132 | (num-ops size-t) |
271 | 133 | (success (array unsigned-char)) |
134 | (new-vlen (* size-t))) |
|
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
135 | (log:debug! (list key klen existing-val existing-vlen ops ops-length num-ops success new-vlen)) |
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
136 | 1) |
271 | 137 | |
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
138 | (define-alien-callable rocksdb-concat-partial-merge boolean |
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
139 | ((key (array unsigned-char)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
140 | (klen size-t) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
141 | (ops (array (array unsigned-char))) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
142 | (ops-length (* size-t)) |
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
143 | (num-ops size-t) |
271 | 144 | (success (array unsigned-char)) |
145 | (new-vlen (* size-t))) |
|
594
5bd0eb9fa1fa
rocksdb callbacks, missing symbol fixes
Richard Westhaver <ellis@rwest.io>
parents:
271
diff
changeset
|
146 | (log:debug! (list key klen ops ops-length num-ops success new-vlen)) |
597
5b2ca5b2a9db
rocksdb merge-op progress
Richard Westhaver <ellis@rwest.io>
parents:
594
diff
changeset
|
147 | 0) |
598
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
148 | |
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
149 | (define-alien-callable rocksdb-concat-delete-value void |
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
150 | ((state (* t)) |
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
151 | (value c-string) |
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
152 | (value-length size-t)) |
c7f9bfc9570f
rm duplicate files, add rocksdb batch and logger files
Richard Westhaver <ellis@rwest.io>
parents:
597
diff
changeset
|
153 | (values)) |