changeset 679: | 12287fab15d0 |
parent: | 97dd03beda03 |
child: | 5f88b237ce29 |
author: | Richard Westhaver <ellis@rwest.io> |
date: | Thu, 26 Sep 2024 21:16:45 -0400 |
permissions: | -rw-r--r-- |
description: | rocksdb load opts and env updates |
18 | 1 | ;;; rdb.lisp --- High-level RocksDB API |
2 | ||
3 | ;; a thin ORM for working with RocksDB storage. |
|
4 | ||
5 | ;; Low-level bindings are in rocksdb.lisp. |
|
6 | ||
7 | ;; Commentary: |
|
8 | ||
9 | ;; Code: |
|
93 | 10 | (defpackage :rdb |
491
298ca41f7f5a
rdb/query updates, clap fixes
Richard Westhaver <ellis@rwest.io>
parents:
371
diff
changeset
|
11 | (:use :cl :std :rocksdb :sb-alien :obj/db :obj/query) |
18 | 12 | (:import-from :sb-ext :string-to-octets :octets-to-string) |
13 | (:export |
|
93 | 14 | ;; err |
98 | 15 | :with-errptr |
99 | 16 | :rdb-error |
102 | 17 | :open-db-error |
113
afcb1f02bb66
removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents:
112
diff
changeset
|
18 | :open-backup-engine-error |
102 | 19 | :destroy-db-error |
113
afcb1f02bb66
removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents:
112
diff
changeset
|
20 | :destroy-backup-engine-error |
99 | 21 | :rocksdb-error |
22 | :rdb-user-error |
|
93 | 23 | :put-kv-error |
24 | :get-kv-error |
|
99 | 25 | :handle-errptr |
113
afcb1f02bb66
removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents:
112
diff
changeset
|
26 | ;; raw |
99 | 27 | :make-rocksdb-options :default-rocksdb-options |
93 | 28 | :open-db-raw :with-open-db-raw |
29 | :close-db-raw :destroy-db-raw |
|
30 | :put-kv-raw :put-kv-str-raw |
|
31 | :get-kv-raw :get-kv-str-raw |
|
32 | :put-cf-raw :put-cf-str-raw |
|
33 | :get-cf-raw :get-cf-str-raw |
|
110
cae8da4b1415
rocksdb macrofication, fixes for RDB (C Strings will be the death of me), removed PWN - quicklisp package conflict, smh.
ellis <ellis@rwest.io>
parents:
102
diff
changeset
|
34 | :create-cf-raw |
111 | 35 | :create-iter :with-iter-raw |
36 | :iter-key-raw :iter-key-str-raw |
|
37 | :iter-val-raw :iter-val-str-raw |
|
112 | 38 | :open-backup-engine-raw |
39 | :close-backup-engine-raw |
|
40 | :create-new-backup-raw |
|
41 | :restore-from-latest-backup-raw |
|
42 | :with-open-backup-engine-raw |
|
292
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
43 | :create-snapshot-raw :release-snapshot-raw |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
44 | :create-sst-writer-raw :finish-sst-writer-raw |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
45 | :destroy-sst-writer-raw :open-sst-writer-raw |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
46 | :sst-put-raw :sst-delete-raw :sst-delete-range-raw :sst-file-size-raw |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
47 | :sst-put-str-raw |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
48 | :open-sst-file :close-sst-file |
93 | 49 | ;; proto |
293
e2e5c4831389
std fixes and rdb with-sst
Richard Westhaver <ellis@rwest.io>
parents:
292
diff
changeset
|
50 | :find-cf |
94 | 51 | :put-key :put-kv |
52 | :get-key :get-kv |
|
53 | :put-cf-key :get-cf-key |
|
54 | :push-sap :push-sap* |
|
269 | 55 | :pull-sap :pull-sap* |
273
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
56 | :backfill-opts :push-opts |
94 | 57 | :get-opt :set-opt |
225 | 58 | :push-cf :open-cfs |
59 | :create-cf :create-cfs |
|
94 | 60 | :insert-kv :insert-key |
113
afcb1f02bb66
removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents:
112
diff
changeset
|
61 | :open-db :close-db :destroy-db |
272
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
62 | :get-prop :get-metadata |
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
63 | :multi-get :delete-key |
292
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
64 | :delete-key-ts :put-key-ts |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
65 | :delete-key-range |
272
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
66 | :make-transaction :commit-transaction |
273
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
67 | :begin-transaction :prepare-transaction |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
68 | :rollback-transaction :delete-transaction |
272
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
69 | :flush-db :flush-cf |
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
70 | :repair-db :write-db |
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
71 | :backup-db :restore-db |
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
72 | :get-stats :snapshot-db |
273
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
73 | :shutdown-db :print-stats |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
74 | :create-iter :iter-next |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
75 | :iter-prev :iter-seek |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
76 | :iter-key :iter-val |
282 | 77 | :iter-timestamp :iter-kv |
78 | :iter-seek-to-first |
|
79 | :iter-seek-to-last |
|
80 | :iter-seek-for-prev |
|
81 | :iter-valid-p |
|
292
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
82 | :ingest-db |
113
afcb1f02bb66
removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents:
112
diff
changeset
|
83 | ;; sst |
292
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
84 | :sst-file-writer :make-sst-file-writer |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
85 | :open-sst :finish-sst |
00d1c8afcdbb
mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents:
282
diff
changeset
|
86 | :destroy-sst :sst-file-size |
93 | 87 | ;; obj |
111 | 88 | :rdb :make-rdb :create-db |
93 | 89 | :rdb-db :rdb-name :rdb-cfs :rdb-opts |
273
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
90 | :rdb-cf-metadata :make-rdb-cf-metadata |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
91 | :rdb-cf-metadata-name :rdb-cf-metadata-size |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
92 | :rdb-level-metadata :make-rdb-level-metadata |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
93 | :rdb-sst-file-metadata :make-rdb-sst-file-metadata |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
94 | :rdb-sst-file-metadata-p :rdb-cf-metadata-p |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
95 | :rdb-level-metadata-p |
111 | 96 | :rdb-bytes :rdb-bytes-buffer :rdb-opts-sap |
267
f3d814fb136a
db upgrades, alik, ignoring c files (for now)
Richard Westhaver <ellis@rwest.io>
parents:
225
diff
changeset
|
97 | :make-key :make-kv :make-val :rdb-kv :rdb-key :rdb-val |
282 | 98 | :rdb-kv |
93 | 99 | :rdb-opts :make-rdb-opts |
100 | :default-rdb-opts |
|
101 | :rdb-cf :make-rdb-cf :create-cf |
|
102 | :rdb-cf-sap :rdb-cf-name |
|
273
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
103 | :rdb-iter :make-rdb-iter :rdb-iter-p |
6d56c4950fa2
rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents:
272
diff
changeset
|
104 | :rdb-iter-sap |
93 | 105 | ;; macs |
272
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
106 | :with-errptr |
93 | 107 | :with-db |
274
5f782d361e08
threads and db tweaks. fixed a tricky macro error caused by string-case, dat/html now works.
Richard Westhaver <ellis@rwest.io>
parents:
273
diff
changeset
|
108 | :*temp-db-destroy* |
272
0a5e37693fdf
db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents:
269
diff
changeset
|
109 | :with-temp-db |
269 | 110 | :do-db |
267
f3d814fb136a
db upgrades, alik, ignoring c files (for now)
Richard Westhaver <ellis@rwest.io>
parents:
225
diff
changeset
|
111 | :with-cf |
269 | 112 | :do-cf |
113 | :with-iter ;; generic |
|
293
e2e5c4831389
std fixes and rdb with-sst
Richard Westhaver <ellis@rwest.io>
parents:
292
diff
changeset
|
114 | :do-cfs |
624
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
115 | :with-sst |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
116 | :nil |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
117 | :rdb-cf-p |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
118 | :copy-rdb-cf |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
119 | :rdb-cf-key-type |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
120 | :rdb-cf-val-type |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
121 | :close-cf |
97dd03beda03
rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents:
491
diff
changeset
|
122 | :open-cf |
679
12287fab15d0
rocksdb load opts and env updates
Richard Westhaver <ellis@rwest.io>
parents:
624
diff
changeset
|
123 | :close-cfs |
12287fab15d0
rocksdb load opts and env updates
Richard Westhaver <ellis@rwest.io>
parents:
624
diff
changeset
|
124 | :rdb-cf-opts |
12287fab15d0
rocksdb load opts and env updates
Richard Westhaver <ellis@rwest.io>
parents:
624
diff
changeset
|
125 | :with-latest-opts)) |
150 | 126 | |
127 | (in-package :rdb) |
|
371 | 128 | (rocksdb:load-rocksdb nil) |