changelog shortlog graph tags branches changeset files file revisions raw help

Mercurial > core / annotate lisp/lib/rdb/pkg.lisp

changeset 679: 12287fab15d0
parent: 97dd03beda03
child: 5f88b237ce29
author: Richard Westhaver <ellis@rwest.io>
date: Thu, 26 Sep 2024 21:16:45 -0400
permissions: -rw-r--r--
description: rocksdb load opts and env updates
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
1
 ;;; rdb.lisp --- High-level RocksDB API
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
2
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
3
 ;; a thin ORM for working with RocksDB storage. 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
4
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
5
 ;; Low-level bindings are in rocksdb.lisp.
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
6
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
7
 ;; Commentary:
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
8
 
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
9
 ;; Code:
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
10
 (defpackage :rdb
491
298ca41f7f5a rdb/query updates, clap fixes
Richard Westhaver <ellis@rwest.io>
parents: 371
diff changeset
11
   (:use :cl :std :rocksdb :sb-alien :obj/db :obj/query)
18
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
12
   (:import-from :sb-ext :string-to-octets :octets-to-string)
61482ce290f9 migration complete
ellis <ellis@rwest.io>
parents:
diff changeset
13
   (:export 
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
14
    ;; err
98
91a382b59e5c lisp test fixes
ellis <ellis@rwest.io>
parents: 97
diff changeset
15
    :with-errptr
99
40d91ba5a115 rdb stuff
ellis <ellis@rwest.io>
parents: 98
diff changeset
16
    :rdb-error
102
db52ddb25d7f progress
ellis <ellis@rwest.io>
parents: 99
diff changeset
17
    :open-db-error
113
afcb1f02bb66 removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents: 112
diff changeset
18
    :open-backup-engine-error
102
db52ddb25d7f progress
ellis <ellis@rwest.io>
parents: 99
diff changeset
19
    :destroy-db-error
113
afcb1f02bb66 removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents: 112
diff changeset
20
    :destroy-backup-engine-error
99
40d91ba5a115 rdb stuff
ellis <ellis@rwest.io>
parents: 98
diff changeset
21
    :rocksdb-error
40d91ba5a115 rdb stuff
ellis <ellis@rwest.io>
parents: 98
diff changeset
22
    :rdb-user-error
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
23
    :put-kv-error
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
24
    :get-kv-error
99
40d91ba5a115 rdb stuff
ellis <ellis@rwest.io>
parents: 98
diff changeset
25
    :handle-errptr
113
afcb1f02bb66 removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents: 112
diff changeset
26
    ;; raw
99
40d91ba5a115 rdb stuff
ellis <ellis@rwest.io>
parents: 98
diff changeset
27
    :make-rocksdb-options :default-rocksdb-options
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
28
    :open-db-raw :with-open-db-raw
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
29
    :close-db-raw :destroy-db-raw
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
30
    :put-kv-raw :put-kv-str-raw
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
31
    :get-kv-raw :get-kv-str-raw
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
32
    :put-cf-raw :put-cf-str-raw
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
33
    :get-cf-raw :get-cf-str-raw
110
cae8da4b1415 rocksdb macrofication, fixes for RDB (C Strings will be the death of me), removed PWN - quicklisp package conflict, smh.
ellis <ellis@rwest.io>
parents: 102
diff changeset
34
    :create-cf-raw
111
2aec94d6a480 rdb fixes
ellis <ellis@rwest.io>
parents: 110
diff changeset
35
    :create-iter :with-iter-raw
2aec94d6a480 rdb fixes
ellis <ellis@rwest.io>
parents: 110
diff changeset
36
    :iter-key-raw :iter-key-str-raw
2aec94d6a480 rdb fixes
ellis <ellis@rwest.io>
parents: 110
diff changeset
37
    :iter-val-raw :iter-val-str-raw
112
430e69339ac3 fixed rdb tests
ellis <ellis@rwest.io>
parents: 111
diff changeset
38
    :open-backup-engine-raw
430e69339ac3 fixed rdb tests
ellis <ellis@rwest.io>
parents: 111
diff changeset
39
    :close-backup-engine-raw
430e69339ac3 fixed rdb tests
ellis <ellis@rwest.io>
parents: 111
diff changeset
40
    :create-new-backup-raw
430e69339ac3 fixed rdb tests
ellis <ellis@rwest.io>
parents: 111
diff changeset
41
    :restore-from-latest-backup-raw
430e69339ac3 fixed rdb tests
ellis <ellis@rwest.io>
parents: 111
diff changeset
42
    :with-open-backup-engine-raw
292
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
43
    :create-snapshot-raw :release-snapshot-raw
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
44
    :create-sst-writer-raw :finish-sst-writer-raw
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
45
    :destroy-sst-writer-raw :open-sst-writer-raw
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
46
    :sst-put-raw :sst-delete-raw :sst-delete-range-raw :sst-file-size-raw
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
47
    :sst-put-str-raw
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
48
    :open-sst-file :close-sst-file
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
49
    ;; proto
293
e2e5c4831389 std fixes and rdb with-sst
Richard Westhaver <ellis@rwest.io>
parents: 292
diff changeset
50
    :find-cf
94
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 93
diff changeset
51
    :put-key :put-kv
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 93
diff changeset
52
    :get-key :get-kv
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 93
diff changeset
53
    :put-cf-key :get-cf-key
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 93
diff changeset
54
    :push-sap :push-sap*
269
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 267
diff changeset
55
    :pull-sap :pull-sap*
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
56
    :backfill-opts :push-opts
94
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 93
diff changeset
57
    :get-opt :set-opt
225
Richard Westhaver <ellis@rwest.io>
parents: 214
diff changeset
58
    :push-cf :open-cfs
Richard Westhaver <ellis@rwest.io>
parents: 214
diff changeset
59
    :create-cf :create-cfs
94
01051403700f rdb work
ellis <ellis@rwest.io>
parents: 93
diff changeset
60
    :insert-kv :insert-key
113
afcb1f02bb66 removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents: 112
diff changeset
61
    :open-db :close-db :destroy-db
272
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
62
    :get-prop :get-metadata
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
63
    :multi-get :delete-key
292
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
64
    :delete-key-ts :put-key-ts
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
65
    :delete-key-range
272
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
66
    :make-transaction :commit-transaction
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
67
    :begin-transaction :prepare-transaction
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
68
    :rollback-transaction :delete-transaction
272
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
69
    :flush-db :flush-cf
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
70
    :repair-db :write-db
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
71
    :backup-db :restore-db
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
72
    :get-stats :snapshot-db
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
73
    :shutdown-db :print-stats
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
74
    :create-iter :iter-next
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
75
    :iter-prev :iter-seek
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
76
    :iter-key :iter-val
282
da580c7fe954 upgrades
Richard Westhaver <ellis@rwest.io>
parents: 278
diff changeset
77
    :iter-timestamp :iter-kv
da580c7fe954 upgrades
Richard Westhaver <ellis@rwest.io>
parents: 278
diff changeset
78
    :iter-seek-to-first
da580c7fe954 upgrades
Richard Westhaver <ellis@rwest.io>
parents: 278
diff changeset
79
    :iter-seek-to-last
da580c7fe954 upgrades
Richard Westhaver <ellis@rwest.io>
parents: 278
diff changeset
80
    :iter-seek-for-prev
da580c7fe954 upgrades
Richard Westhaver <ellis@rwest.io>
parents: 278
diff changeset
81
    :iter-valid-p
292
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
82
    :ingest-db
113
afcb1f02bb66 removed obj/uri. may add back new impl later. new file obj/db.lisp and RDB stuff - (sst)
ellis <ellis@rwest.io>
parents: 112
diff changeset
83
    ;; sst
292
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
84
    :sst-file-writer :make-sst-file-writer
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
85
    :open-sst :finish-sst
00d1c8afcdbb mostly done with std refactor, added sst-file-writer to rdb
Richard Westhaver <ellis@rwest.io>
parents: 282
diff changeset
86
    :destroy-sst :sst-file-size
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
87
    ;; obj
111
2aec94d6a480 rdb fixes
ellis <ellis@rwest.io>
parents: 110
diff changeset
88
    :rdb :make-rdb :create-db
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
89
    :rdb-db :rdb-name :rdb-cfs :rdb-opts
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
90
    :rdb-cf-metadata :make-rdb-cf-metadata
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
91
    :rdb-cf-metadata-name :rdb-cf-metadata-size
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
92
    :rdb-level-metadata :make-rdb-level-metadata
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
93
    :rdb-sst-file-metadata :make-rdb-sst-file-metadata
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
94
    :rdb-sst-file-metadata-p :rdb-cf-metadata-p
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
95
    :rdb-level-metadata-p
111
2aec94d6a480 rdb fixes
ellis <ellis@rwest.io>
parents: 110
diff changeset
96
    :rdb-bytes :rdb-bytes-buffer :rdb-opts-sap
267
f3d814fb136a db upgrades, alik, ignoring c files (for now)
Richard Westhaver <ellis@rwest.io>
parents: 225
diff changeset
97
    :make-key :make-kv :make-val :rdb-kv :rdb-key :rdb-val 
282
da580c7fe954 upgrades
Richard Westhaver <ellis@rwest.io>
parents: 278
diff changeset
98
    :rdb-kv
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
99
    :rdb-opts :make-rdb-opts
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
100
    :default-rdb-opts
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
101
    :rdb-cf :make-rdb-cf :create-cf
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
102
    :rdb-cf-sap :rdb-cf-name
273
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
103
    :rdb-iter :make-rdb-iter :rdb-iter-p
6d56c4950fa2 rocksdb iter,metadata,and more
Richard Westhaver <ellis@rwest.io>
parents: 272
diff changeset
104
    :rdb-iter-sap
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
105
    ;; macs
272
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
106
    :with-errptr
93
17b6d1f39506 rdb refactor, skel stuff
ellis <ellis@rwest.io>
parents: 84
diff changeset
107
    :with-db
274
5f782d361e08 threads and db tweaks. fixed a tricky macro error caused by string-case, dat/html now works.
Richard Westhaver <ellis@rwest.io>
parents: 273
diff changeset
108
    :*temp-db-destroy*
272
0a5e37693fdf db protocol stuff, prep for tao, tests, with-temp-db
Richard Westhaver <ellis@rwest.io>
parents: 269
diff changeset
109
    :with-temp-db
269
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 267
diff changeset
110
    :do-db
267
f3d814fb136a db upgrades, alik, ignoring c files (for now)
Richard Westhaver <ellis@rwest.io>
parents: 225
diff changeset
111
    :with-cf
269
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 267
diff changeset
112
    :do-cf
87f503c7a365 more rocksdb
Richard Westhaver <ellis@rwest.io>
parents: 267
diff changeset
113
    :with-iter ;; generic
293
e2e5c4831389 std fixes and rdb with-sst
Richard Westhaver <ellis@rwest.io>
parents: 292
diff changeset
114
    :do-cfs
624
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
115
    :with-sst
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
116
    :nil
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
117
    :rdb-cf-p
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
118
    :copy-rdb-cf
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
119
    :rdb-cf-key-type
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
120
    :rdb-cf-val-type
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
121
    :close-cf
97dd03beda03 rocksdb updates in coordination with infra/scripts/org-graph-db-init.lisp
Richard Westhaver <ellis@rwest.io>
parents: 491
diff changeset
122
    :open-cf
679
12287fab15d0 rocksdb load opts and env updates
Richard Westhaver <ellis@rwest.io>
parents: 624
diff changeset
123
    :close-cfs
12287fab15d0 rocksdb load opts and env updates
Richard Westhaver <ellis@rwest.io>
parents: 624
diff changeset
124
    :rdb-cf-opts
12287fab15d0 rocksdb load opts and env updates
Richard Westhaver <ellis@rwest.io>
parents: 624
diff changeset
125
    :with-latest-opts))
150
ellis <ellis@rwest.io>
parents: 113
diff changeset
126
 
ellis <ellis@rwest.io>
parents: 113
diff changeset
127
 (in-package :rdb)
371
73c8c1297989 no-save libs
Richard Westhaver <ellis@rwest.io>
parents: 293
diff changeset
128
 (rocksdb:load-rocksdb nil)