1.1--- a/.hgignore Tue Dec 12 21:50:58 2023 -0500
1.2+++ b/.hgignore Sun Apr 28 19:49:20 2024 -0400
1.3@@ -0,0 +1,1 @@
1.4+[.]html
1.5\ No newline at end of file
2.1--- /dev/null Thu Jan 01 00:00:00 1970 +0000
2.2+++ b/draft/a-bit-of-risc.org Sun Apr 28 19:49:20 2024 -0400
2.3@@ -0,0 +1,310 @@
2.4+#+title: A Bit of RISC
2.5+#+date: [2024-03-11 Mon]
2.6+I recently picked up [[https://dl.acm.org/doi/10.5555/2462741][Hacker's Delight]] and having a lot of fun with
2.7+it. It's a collection of bit-manipulation tricks collected by hackers
2.8+over many years. You can flip open pretty much anywhere in the book
2.9+and start learn something really cool.
2.10+
2.11+There's something about seeing bit strings and assembly code in a book
2.12+that really catches my attention, this one goes a bit further by even
2.13+describing a complete RISC (Reduced Instruction Set Computer) we can
2.14+implement to play around with the various tricks.
2.15+
2.16+As an exercise and for fun, I'd like to employ some Lisp-fu here and
2.17+implement a small VM specifically designed for mangling bits.
2.18+
2.19+As a fair warning, I'm not a mathematician and I don't write proofs
2.20+often. If I get something wrong or there is a better way of doing
2.21+things, let me know!
2.22+
2.23+You can find most of the code from the book [[https://github.com/hcs0/Hackers-Delight][here]].
2.24+
2.25+* Design
2.26+** Data Representation
2.27+We'll be sticking with a 32-bit word length as recommended in the
2.28+Preface. We will also represent registers as integers whenever
2.29+possible, instead of say a bit-vector. Without going into too much
2.30+detail, it's much more efficient to do bitwise ops on integers
2.31+instead of bit-vectors in Lisp.
2.32+
2.33+We need a minimum of 16 general purpose registers, typically of word
2.34+length, with R0 reserved for a constant 0. To address 16 different
2.35+registers we actually only need 4 bits - 5-bits if we needed 32, 6 for
2.36+64, etc.
2.37+
2.38+Floating-point support and special purpose registers are not required.
2.39+
2.40+** Instructions
2.41+The Hacker's Delight RISC architecture is described in two tables,
2.42+denoted =basic RISC= and =full RISC= respectively.
2.43+
2.44+Most instructions take two source registers =RA= and =RB= with a
2.45+destination register =RT=. The actual general-purpose registers are
2.46+labelled =R0= (containg the constant 0) through =R15=.
2.47+
2.48+A 3-Address machine is assumed and some instructions take 16-bit
2.49+signed or unsigned immediate values - denoted =I= and =Iu=
2.50+respectively.
2.51+
2.52+#+tblname: Basic Instruction Set (basic RISC)
2.53+| Opcode Mnemonic | Operands | Description |
2.54+|-----------------------------------------------------------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------|
2.55+| add,sub,mul,div,divu,rem,remu | RT,RA,RB | RT <- (op RA RB) |
2.56+| addi,muli | RT,RA,I | RT <- (op RA I), I is a 16-bit signed immediate-value |
2.57+| addis | RT,RA,I | RT <- (+ RA (\vert\vert I 0x0000)) |
2.58+| and,or,xor | RT,RA,RB | RT <- (op RA RB) |
2.59+| andi,ori,xori | RT,RA,Iu | As above, expect the last operand is a 16-bit unsigned immediate-value |
2.60+| beq,bne,blt,ble,bgt,bge | RT,target | Branch to target if (op RT) |
2.61+| bt,bf | RT,target | Branch true/false, same as bne/beq resp |
2.62+| cmpeq,cmpne,cmplt,cmple,cmpgt,cmpge,cmpltu,cmpleu,cmpgtu,cmpgeu | RT,RA,RB | RT <- (if (op RA RB) 1 0) |
2.63+| cmpieq,cmpine,cmpilt,cmpile,cmpigt,cmpige | RT,RA,I | Like cmpeq except second comparand is a 16-bit signed immediate-value |
2.64+| cmpiequ,cmpineu,cmpiltu,cmpileu,cmpigtu,cmpigeu | RT,RA,I | Like cmpltu except second comparand is a 16-bit unsigned immediate-value |
2.65+| ldbu,ldh,ldhu,ldw | RT,d(RA) | Load an unsigned-byte, signed-halfword, unsigned-halfword, or word into RT from (+ RA d) where d is a 16-bit signed immediate-value |
2.66+| mulhs,mulhu | RT,RA,RB | RT gets the high-order 32 bits of (* RA RB) |
2.67+| not | RT,RA | RT <- bitwise one's-complement of RA |
2.68+| shl,shr,shrs | RT,RA,RB | RT <- RA shifted left or right by rightmost six bits of RB; 0-fill except for shrs, which is sign-fill (shift amount treated modulo 64) |
2.69+| shli,shri,shrsi | RT,RA,Iu | RT <- RA shifted left or right by 5-bit immediate field |
2.70+| stb,sth,stw | RS,d(RA) | Store a byte,halfword,word from RS into memory at location (+ RA d) where d is a 16-bit signed immediate-value |
2.71+
2.72+
2.73+#+name: Addition Instructions (full RISC)
2.74+| Opcode Mnemonic | Operands | Description |
2.75+|-----------------------------------------------------------------+-----------+--------------------------------------------------------------------------------------------------------|
2.76+| abs,nabs | RT,RA | RT <- (op RA) |
2.77+| andc,eqv,nand,nor,orc | RT,RA,RB | RT <- (op RA RB) |
2.78+| extr | RT,RA,I,L | extract bits I through I+L-1 of RA and place them right-adjusted in RT, with 0-fill |
2.79+| extrs | RT,RA,I,L | Like extr, but sign-fill |
2.80+| ins | RT,RA,I,L | Insert bits 0 through L-1 of RA into bits I through I+L-1 of RT |
2.81+| nlz | RT,RA | RT gets count of leading 0's in RA (0 to 32) |
2.82+| pop | RT,RA | RT gets the number of 1-bits in RA (0 to 32) |
2.83+| ldb | RT,d(RA) | Load a signed byte into RT from memory at location (+ RA d) where d is a 16-bit signed immediate value |
2.84+| moveq,movne,movlt,movle,movgt,movge | RT,RA,RB | RT <- RA rotate-shifted left or right by the rightmost 5-bits of RB |
2.85+| shlr,shrr | RT,RA,RB | RT <- RA rotate-shifted left or right by the rightmost 5-bits of RB |
2.86+| shlri,shrri | RT,RA,Iu | RT <- RA rotate-shifted left or right by the 5-bit immediate field |
2.87+| trpeq,trpne,trplt,trple,trpgt,trpge,trpltu,trpleu,trpgtu,trpgeu | RA,RB | Trap (interrupt) if (op RA RB) |
2.88+| trpieq,trpine,trpilt,trpile,trpigt,trpige | RA,I | Trap if (op RA I) where I is a 16-bit signed immediate-value |
2.89+| trpiequ,trpineu,trpiltu,trpileu,trpigtu,trpigeu | RA,Iu | Trap if (op RA Iu) where Iu is a 16-bit unsigned immediate-value |
2.90+
2.91+There is also some extensions, which are like macros that usually
2.92+expand to a single instruction.
2.93+
2.94+#+name: Extended Mnemonics
2.95+| Extended Mnemonic | Expansion | Description |
2.96+|-------------------+------------------+---------------------------|
2.97+| b target | beq R0,target | Unconditional branch |
2.98+| li RT,I | (addi,addis,ori) | Load immediate |
2.99+| mov RT,RA | ori RT,RA,0 | Move register RA to RT |
2.100+| neg RT,RA | sub RT,R0,RA | Negate (two's-complement) |
2.101+| subi RT,RA,I | addi RT,RA,-I | Subtract immediate |
2.102+
2.103+All of these instructions are available on x86,arm,riscv and the likes
2.104+so no real surprises. We will implement the basic set in Lisp, mapping
2.105+instructions directly to Lisp functions using macros.
2.106+
2.107+** Execution Model
2.108+We'll build this machine in Lisp and use plenty intrinsics from
2.109+SBCL. As a starting point I followed Paul Khuong's excellent blog
2.110+post: [[https://pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembly-code-breadboard/][SBCL: The ultimate assembly code breadboard]].
2.111+
2.112+Some things to keep in mind for our machine:
2.113+- every instruction requires at most two register reads and one
2.114+ register write - good for compilers
2.115+- every instruction counts as a single cycle
2.116+- we pay no attention to instruction-level parallelism
2.117+
2.118+* The HAKMEM VM
2.119+:properties:
2.120+:header-args: :session t :results none
2.121+:end:
2.122+#+name: defpackage
2.123+#+begin_src lisp
2.124+ (ql:quickload :prelude)
2.125+ (in-package :std-user)
2.126+ (defpackage :hakmem
2.127+ (:use :cl :std)
2.128+ (:import-from :sb-assem :inst)
2.129+ (:import-from :sb-vm :immediate-constant :registers :zero :ea))
2.130+ (in-package :hakmem)
2.131+ ;; (in-package :sb-x86-64-asm)
2.132+ (in-readtable :std)
2.133+ (declaim (optimize (speed 3) (safety 1)))
2.134+ (defconstant +word-size+ 32 "default word size and register length.")
2.135+#+end_src
2.136+
2.137+#+name: vars
2.138+#+begin_src lisp :package hakmem
2.139+ (declaim (type (unsigned-byte #.+word-size+) +ro+))
2.140+ (defconstant +r0+ 0 "constant value for register 0")
2.141+ (defvar *stack* (make-array 8 :initial-contents (list sb-vm::r8-tn
2.142+ sb-vm::r9-tn
2.143+ sb-vm::r10-tn
2.144+ sb-vm::r11-tn
2.145+ sb-vm::r12-tn
2.146+ sb-vm::r13-tn
2.147+ sb-vm::r14-tn
2.148+ sb-vm::r15-tn)))
2.149+ (defvar *stack-pointer*)
2.150+
2.151+ (defvar *rax* sb-vm::rax-tn)
2.152+ (defvar *rbx* sb-vm::rax-tn)
2.153+ (defvar *rcx* sb-vm::rax-tn)
2.154+ (defvar *rdx* sb-vm::rax-tn)
2.155+
2.156+ ;; (@ 0) returns the (current) register for TOS, (@ 1) returns
2.157+ ;; the one just below, etc.
2.158+ (defun @ (i)
2.159+ (aref *stack* (mod (+ i *stack-pointer*) (length *stack*))))
2.160+
2.161+ (defvar *code-base* sb-vm::rsi-tn)
2.162+ (defvar *virtual-ip* sb-vm::rdi-tn)
2.163+ (sb-x86-64-asm::get-gpr :qword 4)
2.164+ ;; (sb-vm::immediate-constant-sc 10000)
2.165+ ;; arena vector or list?
2.166+ (defvar *instructions* (make-hash-table :test #'equal))
2.167+
2.168+ (defvar *primitive-code-offset* (* 64 67))
2.169+
2.170+ (defstruct code-page
2.171+ (alloc 0) ;; next free byte
2.172+ (code (make-array *primitive-code-offset* :element-type 'octet)))
2.173+
2.174+ (defun emit-code (pages emitter)
2.175+ ;; there must be as many code pages as there are stack slots
2.176+ (assert (= (length *stack*) (length pages)))
2.177+ ;; find the rightmost starting point, and align to 16 bytes
2.178+ (let* ((alloc (logandc2 (+ 15 (reduce #'max pages :key #'code-page-alloc))
2.179+ 15))
2.180+ (bytes (loop for i below (length pages)
2.181+ for page = (elt pages i)
2.182+ collect (let ((segment (sb-assem:make-segment))
2.183+ (*stack-pointer* i))
2.184+ ;; assemble the variant for this value
2.185+ ;; of *stack-pointer* in a fresh code
2.186+ ;; segment
2.187+ (sb-assem:assemble (segment)
2.188+ ;; but first, insert padding
2.189+ (sb-vm::emit-long-nop segment (- alloc (code-page-alloc page)))
2.190+ (funcall emitter))
2.191+ ;; tidy up any backreference
2.192+ (sb-assem:finalize-segment segment)
2.193+ ;; then get the (position-independent) machine
2.194+ ;; code as a vector of bytes
2.195+ (sb-assem:segment-contents-as-vector segment)))))
2.196+ ;; finally, copy each machine code sequence to the right code page
2.197+ (map nil (lambda (page bytes)
2.198+ (let ((alloc (code-page-alloc page)))
2.199+ (replace (code-page-code page) bytes :start1 alloc)
2.200+ (assert (<= (+ alloc (length bytes)) (length (code-page-code page))))
2.201+ (setf (code-page-alloc page) (+ alloc (length bytes)))))
2.202+ pages bytes)
2.203+ ;; and return the offset for that code sequence
2.204+ alloc))
2.205+
2.206+ (defun emit-all-code (&rest emitters)
2.207+ (let ((pages (loop repeat (length *stack*)
2.208+ for page = (make-code-page)
2.209+ ;; prefill everything with one-byte NOPs
2.210+ do (fill (code-page-code page) #x90)
2.211+ collect page)))
2.212+ (values (mapcar (lambda (emitter)
2.213+ (emit-code pages emitter))
2.214+ emitters)
2.215+ pages)))
2.216+
2.217+ (defun next (&optional offset)
2.218+ (setf offset (or offset 0)) ; accommodate primops that frob IP
2.219+ (let ((rotation (mod *stack-pointer* (length *stack*))))
2.220+ (inst movzx *rax* (make-ea :dword :base *virtual-ip*
2.221+ :disp offset))
2.222+ (unless (= -4 offset)
2.223+ (inst add *virtual-ip* (+ 4 offset)))
2.224+ (if (zerop rotation)
2.225+ (inst add *rax* *code-base*)
2.226+ (inst lea *rax* (make-ea :qword :base *code-base*
2.227+ :index *rax*
2.228+ :disp (* rotation *primitive-code-offset*))))
2.229+ (inst jmp *rax*)))
2.230+
2.231+ (defun swap ()
2.232+ (inst xchg (@ 0) (@ 1)) ; exchange top of stack and stack[1]
2.233+ (next))
2.234+
2.235+#+end_src
2.236+
2.237+#+name: instructions
2.238+#+begin_src lisp :package hakmem
2.239+ ;; todo
2.240+ (defun %parse-reg3 (rt ra rb))
2.241+ (defun %parse-reg2i (rt ra i))
2.242+ (defun %parse-reg2ui (rt ra ui))
2.243+ (defmacro def-inst (name args &body body)
2.244+ ;; todo: compose a function based on regs+args+body
2.245+ `(let ((sc *scratch*)
2.246+ (r0 +r0+)
2.247+ (ra 0)
2.248+ (rb 0)
2.249+ (rt 0))
2.250+ (declare (ignorable sc r0 ra rb rt))
2.251+ (setf (gethash ',name *instructions*) (lambda ,args (progn ,@body)))))
2.252+
2.253+ (defmacro def-prim (name op)
2.254+ `(def-inst ,name () (setf rt (,op ra rb))))
2.255+#+end_src
2.256+
2.257+#+name: prims
2.258+#+begin_src lisp :package hakmem
2.259+ (def-prim add +)
2.260+ (def-prim sub -)
2.261+ (def-prim mul *)
2.262+ (def-prim div /)
2.263+ ;; divu
2.264+ (def-prim rem mod)
2.265+ ;; remu
2.266+ (def-prim cmpeq =)
2.267+ (def-prim cmpne /=)
2.268+ (def-prim cmplt <)
2.269+ (def-prim cmple <=)
2.270+ (def-prim cmpgt >)
2.271+ (def-prim cmpge >=)
2.272+ ;; ltu leu gtu geu
2.273+ (def-inst addi (i)
2.274+ (setf rt (+ ra i)))
2.275+ (def-inst muli (i)
2.276+ (setf rt (* ra i)))
2.277+ (def-prim and logand)
2.278+ (def-prim or logior)
2.279+ (def-prim xor logxor)
2.280+
2.281+ (defun get-inst (i) (gethash i *instructions*))
2.282+
2.283+ (defmacro %inst (i &body args)
2.284+ `(funcall (get-inst ',i) ,@args))
2.285+
2.286+ (defun list-instructions (&optional (tbl *instructions*))
2.287+ (hash-table-alist tbl))
2.288+
2.289+#+end_src
2.290+#+name: instruction-list
2.291+#+begin_src lisp :results replace :exports both :package hakmem
2.292+ (list-instructions)
2.293+#+end_src
2.294+
2.295+#+RESULTS: instruction-list
2.296+#+begin_example
2.297+((XOR . #<FUNCTION (LAMBDA ()) {10077C774B}>)
2.298+ (OR . #<FUNCTION (LAMBDA ()) {10077C777B}>)
2.299+ (AND . #<FUNCTION (LAMBDA ()) {10077C77AB}>)
2.300+ (MULI . #<FUNCTION (LAMBDA (I)) {10077C77DB}>)
2.301+ (ADDI . #<FUNCTION (LAMBDA (I)) {10077C780B}>)
2.302+ (CMPGE . #<FUNCTION (LAMBDA ()) {10077C783B}>)
2.303+ (CMPGT . #<FUNCTION (LAMBDA ()) {10077C786B}>)
2.304+ (CMPLE . #<FUNCTION (LAMBDA ()) {10077C789B}>)
2.305+ (CMPLT . #<FUNCTION (LAMBDA ()) {10077C78CB}>)
2.306+ (CMPNE . #<FUNCTION (LAMBDA ()) {10077C78FB}>)
2.307+ (CMPEQ . #<FUNCTION (LAMBDA ()) {10077C792B}>)
2.308+ (REM . #<FUNCTION (LAMBDA ()) {10077C795B}>)
2.309+ (DIV . #<FUNCTION (LAMBDA ()) {10077C79AB}>)
2.310+ (MUL . #<FUNCTION (LAMBDA ()) {10077C79EB}>)
2.311+ (SUB . #<FUNCTION (LAMBDA ()) {10077C7A1B}>)
2.312+ (ADD . #<FUNCTION (LAMBDA ()) {10077C7A4B}>))
2.313+#+end_example
3.1--- /dev/null Thu Jan 01 00:00:00 1970 +0000
3.2+++ b/draft/a-lispy-database.org Sun Apr 28 19:49:20 2024 -0400
3.3@@ -0,0 +1,25 @@
3.4+#+title: A Lispy Database
3.5+#+options: toc:t h:1
3.6+One of the key features missing in the =core= is a DBMS. The first,
3.7+and often only choice for this need in companies of today is
3.8+a SQL RDBMS.
3.9+
3.10+There are client-server systems like PostgreSQL and MySQL, and
3.11+embedded offerings like SQLite. Whatever lang you use you can count on
3.12+there being SQL support[fn:1]. To support all the different SQL
3.13+flavors though, a new abstraction has been summoned - the ORM.
3.14+
3.15+The ORM maps the object system of your lang to SQL tables, columns and
3.16+rows and provides an API for you to manipulate the database in a more
3.17+idiomatic way.
3.18+
3.19+You might ask yourself, why choose anything else? For good reason. We
3.20+all know friendly neighborhood SQL and understand it. Many of us have
3.21+never used anything else because we don't /need/ to. You can represent
3.22+fairly complex relationships in the RDBMS model and have an
3.23+ecosystem that supports it wherever you go.
3.24+
3.25+My answer is that just because SQL is good enough, doesn't mean it's
3.26+always the best choice.
3.27+
3.28+[fn:1]
4.1--- /dev/null Thu Jan 01 00:00:00 1970 +0000
4.2+++ b/draft/dylib-skel.org Sun Apr 28 19:49:20 2024 -0400
4.3@@ -0,0 +1,146 @@
4.4+#+title: Shared Library Skeletons
4.5+* Overview
4.6++ CODE :: [[https://lab.rwest.io/packy/stash/dysk][packy/stash/dysk]]
4.7+Our core languages are [[https://www.rust-lang.org/][Rust]] and [[https://lisp-lang.org/][Lisp]] - this is the killer combo which will allow NAS-T
4.8+to rapidly develop high-quality software. As such, it's crucial that these two very
4.9+different languages (i.e. compilers) are able to interoperate seamlessly.
4.10+
4.11+Some interop methods are easy to accomodate via the OS - such as IPC or data sharing,
4.12+but others are a bit more difficult.
4.13+
4.14+In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something
4.15+that /can/ be difficult, due to some complications with Rust and because this is not the
4.16+most popular software stack (yet ;). This is an experiment and may not make it to our
4.17+code-base, but it's definitely something worth adding to the toolbox in case we need it.
4.18+** FFI
4.19+The level of interop we're after in this case is [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFI]].
4.20+
4.21+Basically, calling Rust code from Lisp and vice-versa. There's an article about calling
4.22+Rust from Common Lisp [[https://dev.to/veer66/calling-rust-from-common-lisp-45c5][here]] which shows the basics and serves as a great starting point
4.23+for those interested.
4.24+*** Rust ABI
4.25+The complication(s) with Rust I mentioned early is really just that /it is not C/. =C=
4.26+is old, i.e. well-supported with a stable ABI, making the process of creating bindings
4.27+for a C library a breeze in many languages.
4.28+
4.29+For a Rust library we need to first appease the compiler, as explained in [[https://doc.rust-lang.org/nomicon/ffi.html#calling-rust-code-from-c][this section]]
4.30+of the Rustonomicon. Among other things it involves changing the calling-convention of
4.31+functions with a type signature and editing the Cargo.toml file to produce a
4.32+C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used
4.33+like the C ABI can.
4.34+
4.35+[[https://github.com/rodrimati1992/abi_stable_crates][abi_stable_crates]] is a project which addresses some of the ABI concerns, presenting a
4.36+sort of ABI-API as a Rust library. Perhaps this is the direction the ecosystem will go
4.37+with in order to maintain an unstable ABI, but for now there is no 'clear' pathway for a
4.38+friction-less FFI development experience in Rust.
4.39+
4.40+*** Overhead
4.41+Using FFI involves some overhead. Check [[https://github.com/dyu/ffi-overhead][here]] for an example benchmark across a few
4.42+languages. While building the NAS-T core, I'm very much aware of this, and will need a
4.43+few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular,
4.44+I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp).
4.45+
4.46+* basic example
4.47+** Setup
4.48+For starters, I'm going to assume we all have Rust (via [[https://rustup.rs/][rustup]]) and Lisp ([[https://www.sbcl.org/][sbcl]] only)
4.49+installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in
4.50+this post).
4.51+*** Cargo
4.52+Create a new library crate. For this example we're focusing on a 'skeleton' for
4.53+/dynamic/ libraries only, so our experiment will be called =dylib-skel= or *dysk* for
4.54+short.
4.55+src_sh[:exports code]{cargo init dysk --lib && cd dysk}
4.56+
4.57+A =src/lib.rs= will be generated for you. Go ahead and delete that. We're going to be
4.58+making our own =lib.rs= file directly in the root directory (just to be cool).
4.59+
4.60+The next step is to edit your =Cargo.toml= file. Add these lines after the =[package]=
4.61+section and before =[dependencies]=:
4.62+#+begin_src conf-toml
4.63+[lib]
4.64+crate-type = ["cdylib","rlib"]
4.65+path = "lib.rs"
4.66+[[bin]]
4.67+name="dysk-test"
4.68+path="test.rs"
4.69+#+end_src
4.70+
4.71+This tells Rust to generate a shared C-compatible object with a =.so= extension which we
4.72+can open using [[https://man.archlinux.org/man/dlopen.3.en][dlopen]].
4.73+*** cbindgen
4.74+**** install
4.75+Next, we want the =cbindgen= program which we'll use to generate header files for
4.76+C/C++. This step isn't necessary at all, we just want it for further experimentation.
4.77+
4.78+src_sh[:exports code]{cargo install --force cbindgen}
4.79+
4.80+We append the =cbindgen= crate as a /build dependency/ to our =Cargo.toml= like so:
4.81+#+begin_src conf-toml
4.82+[build-dependencies]
4.83+cbindgen = "0.24"
4.84+#+end_src
4.85+**** cbindgen.toml
4.86+#+begin_src conf-toml :tangle cbindgen.toml
4.87+language = "C"
4.88+autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */"
4.89+include_version = true
4.90+namespace = "dysk"
4.91+cpp_compat = true
4.92+after_includes = "#define DYSK_VERSION \"0.1.0\""
4.93+line_length = 88
4.94+tab_width = 2
4.95+documentation = true
4.96+documentation_style = "c99"
4.97+usize_is_size_t = true
4.98+[cython]
4.99+header = "dysk.h"
4.100+#+end_src
4.101+**** build.rs
4.102+#+begin_src rust :tangle build.rs
4.103+fn main() -> Result<(), cbindgen::Error> {
4.104+ if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) {
4.105+ b.write_to_file("dysk.h"); Ok(())}
4.106+ else { panic!("failed to generate dysk.h from cbindgen.toml") } }
4.107+#+end_src
4.108+** lib.rs
4.109+#+begin_src rust :tangle lib.rs
4.110+//! lib.rs --- dysk library
4.111+use std::ffi::{c_char, c_int, CString};
4.112+#[no_mangle]
4.113+pub extern "C" fn hello() -> *const c_char {
4.114+ CString::new("hello from rust").unwrap().into_raw()}
4.115+#[no_mangle]
4.116+pub extern "C" fn plus(a:c_int,b:c_int) -> c_int {a+b}
4.117+#[no_mangle]
4.118+pub extern "C" fn plus1(n:c_int) -> c_int {n+1}
4.119+#+end_src
4.120+** test.rs
4.121+#+begin_src rust :tangle test.rs
4.122+//! test.rs --- dysk test
4.123+fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::plus1(2 as core::ffi::c_int);}}
4.124+#+end_src
4.125+** compile
4.126+#+begin_src sh
4.127+cargo build --release
4.128+#+end_src
4.129+** load from SBCL
4.130+#+begin_src lisp :tangle dysk.lisp
4.131+;;; dysk.lisp
4.132+;; (dysk:hello) ;; => "hello from rust"
4.133+(defpackage :dysk
4.134+ (:use :cl :sb-alien)
4.135+ (:export :hello :plus :plus1))
4.136+(in-package :dysk)
4.137+(load-shared-object #P"target/release/libdysk.so")
4.138+(define-alien-routine hello c-string)
4.139+(define-alien-routine plus int (a int) (b int))
4.140+(define-alien-routine plus1 int (n int))
4.141+#+end_src
4.142+** benchmark
4.143+#+begin_src shell
4.144+time target/release/dysk-test
4.145+#+end_src
4.146+#+begin_src lisp :tangle test.lisp
4.147+(time (dotimes (_ 500000000) (dysk:plus1 2)))
4.148+#+end_src
4.149+
5.1--- /dev/null Thu Jan 01 00:00:00 1970 +0000
5.2+++ b/draft/hello-world.org Sun Apr 28 19:49:20 2024 -0400
5.3@@ -0,0 +1,88 @@
5.4+#+title: (hello world)
5.5+#+options: toc:t h:1
5.6+* COMMENT Introduction
5.7+Hello World,
5.8+
5.9+I've worked in and around software systems for many years. Throughout
5.10+my journey I've worked with some incredibly capable programmers,
5.11+leaders, sales professionals, and curious minds from all walks of
5.12+life. I've been fortunate to witness some highly effective teams on
5.13+complex projects deliver under impossible constraints. I've also seen
5.14+the other side - where a team isn't effective for one reason or
5.15+another, even on simple problems.
5.16+
5.17+My work as of late is stemmed from the simple premise that there
5.18+exists an /environment/ in which a team can be highly effective. My
5.19+goal is to find those local optimums in my areas of interest where
5.20+such an environment can be built and /great work/ can be done.
5.21+
5.22+In my professional experience, it is becoming much more difficult to
5.23+build an /environment/ where people can be effective. There are
5.24+several contributing factors which muddy the waters but it all boils
5.25+down to capabilities and politics.
5.26+
5.27+Your environment must be capable. It must provide everything your team
5.28+needs now and /may/ need in the future. If the environment doesn't
5.29+provide something, you need to provide all building materials for it -
5.30+/even if you don't know what you're building yet/.
5.31+
5.32+To build a capable environment you need to discard politics from your
5.33+decision-making process. In other words, drop the ego. This requires a
5.34+high degree of introspection. It's hard enough for individuals, let
5.35+alone an entire team or company. In the world of software we tend to
5.36+have two camps - the pro-dev who prefers ergonomic but proprietary
5.37+tools and the foss-dev who prefers clunky but open-source
5.38+alternatives. You can't limit your environment based on the camp you
5.39+align with.
5.40+
5.41+* COMMENT The Compiler Company
5.42+Without further ado, I'd like to announce /The Compiler Company,
5.43+LLC/. The purpose of /The Compiler Company/ is to /compile/
5.44+/companies/.
5.45+
5.46+More specifically, I'm writing a software suite which specializes in
5.47+building environments.
5.48+
5.49+The software isn't for everyone - modules will be rewritten
5.50+frequently, code may be terse in places, with specialized tools,
5.51+custom compilers, and advanced hardware features. It's for a specific
5.52+type of programmer - an /operator/ if you will, who may use it for
5.53+rapid-development of their own programs (or companies). The barrier to
5.54+entry is high.
5.55+
5.56+At this stage, I'm interested in *systems*, *processes*, and
5.57+*protocols* - not *products* and *services*. /The Compiler Company/'s
5.58+most valuable assett its ideas. A /demo/ system is being written which
5.59+serves as a reference implementation but this is currently designed
5.60+for internal benchmarking.
5.61+
5.62+** [[https://compiler.company/docs/core][core]]
5.63+The =core= is a collection of applications and libraries built from
5.64+the bottom-up with modularity in mind. It's primarily written in
5.65+Common Lisp and Rust with minimal external dependencies.
5.66+
5.67+*Lisp* is a first-class citizen of our internal environment. We
5.68+currently rely on the Steel Bank Common Lisp compiler but even if we
5.69+switch to a different implementation there will always be Lisp. It's
5.70+our dedicated high-level programming language (and much more, as we'll
5.71+explain later).
5.72+
5.73+*Rust* is second-class. It meets an arbitrary criteria for what I
5.74+would consider /good enough/ but there are many contenders including
5.75+C, C++, and Zig. It helps fill the gaps in our Lisp environment which
5.76+would be extremely difficult to implement from scratch like
5.77+eliminating GC, compile-time type safety, cross-compilation features,
5.78+and advanced networking protocols. The community support and my
5.79+personal experience with the language are also contributing
5.80+factors. The trade-off is that we need to support another language
5.81+environment in parallel to Lisp.
5.82+
5.83+** [[https://compiler.company/docs/infra][infra]]
5.84+Unfortunately, ideas can't host themselves. We need a robust
5.85+infrastructure to compensate for this. The project =infra= contains
5.86+scripts for building and maintaing the entire corporate
5.87+infrastructure.
5.88+
5.89+We typically host services on Arch Linux. Podman and QEMU are used for
5.90+virtualization. Modules can be built and deployed separately to make
5.91+host-migration easier as we expand.
7.1--- /dev/null Thu Jan 01 00:00:00 1970 +0000
7.2+++ b/draft/outlines.org Sun Apr 28 19:49:20 2024 -0400
7.3@@ -0,0 +1,152 @@
7.4+#+title: outlines
7.5+* Overview
7.6+Source code files are hard to manage. They can get unwieldly quickly and making the
7.7+wrong assumption about your whereabouts in the code tree can have unintended
7.8+consequences.
7.9+
7.10+There are many ways to solve this problem to different degrees. We'll be talking about
7.11+one strategy in particular which I use and recommend for any software project.
7.12+
7.13+Looking through the source code in the NAS-T repository you'll find some common
7.14+commenting patterns:
7.15+
7.16+- every file start with at least one comment line for example:
7.17+#+begin_src lisp
7.18+;;; file-name.lisp --- file description
7.19+#+end_src
7.20+
7.21+- Before you see any code in a file, you'll likely encounter this line:
7.22+#+begin_src lisp
7.23+;;; Code:
7.24+#+end_src
7.25+
7.26+- etc
7.27+
7.28+What's the deal here? To be clear, I'm of the mind that comments should be
7.29+significant. They should express to the reader something that is of a non-trivial nature
7.30+and 'where the code starts' doesn't quite qualify. Indeed, these comments don't fit that
7.31+model at all.
7.32+
7.33+The deal is that these comments aren't for the reader, they're for the developer. More
7.34+specifically, for the developer to treat as a special meta-language to describe the
7.35+structure of a source code file.
7.36+
7.37+* Outlines
7.38+Like all my good ideas, this one is credited entirely to Emacs. In this case, the
7.39+excellent [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Outline-Mode.html][Outline mode]]. If you are an Emacs user you've probably already used it without
7.40+knowing -- Org mode, for example, is [[https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/org.el?h=release_9.6.9#n4789][derived from outline-mode]].
7.41+
7.42+I've grown quite fond of it. Here's the summary:
7.43+
7.44+#+begin_quote
7.45+Outline mode is a major mode derived from Text mode, which is specialized for editing
7.46+outlines. It provides commands to navigate between entries in the outline structure, and
7.47+commands to make parts of a buffer temporarily invisible, so that the outline structure
7.48+may be more easily viewed.
7.49+#+end_quote
7.50+-- [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Outline-Mode.html][GNU]]
7.51+
7.52+** Quickstart
7.53+If you want to jump in right away, I recommend using these keybinds in Emacs:
7.54+
7.55+#+tblname: outline-keys
7.56+| <backtab> | outline-cycle-buffer |
7.57+| M-TAB | outline-cycle |
7.58+| M-n | outline-next-visible-heading |
7.59+| M-p | outline-previous-visible-heading |
7.60+
7.61+Here's a snippet which will enable the keybinds I use:
7.62+
7.63+#+name: enable-outline-keys
7.64+#+begin_src emacs-lisp
7.65+(let ((keys
7.66+ '(("<backtab>" #'outline-cycle-buffer)
7.67+ ("M-TAB" #'outline-cycle)
7.68+ ("M-n" #'outline-next-visible-heading)
7.69+ ("M-p" #'outline-previous-visible-heading))))
7.70+ (cl-loop for (k fn) in keys
7.71+ do (keymap-set outline-minor-mode-map k fn)))
7.72+#+end_src
7.73+
7.74+Now open a file in the [[../../src/][src]] directory, like [[../../src/fs/btrfs/btrfs.lisp][this]] one, enable =outline-minor-mode= and
7.75+move around the file with the new keybinds above.
7.76+
7.77+** Outlines4All
7.78+Not all programming modes have outline support built-in. The good news is that it's easy
7.79+to enable it.
7.80+
7.81+You only need to modify one variable: =outline-regexp= and enable a minor-mode:
7.82+=outline-minor-mode=.
7.83+
7.84+*** Using dir-locals
7.85+The way it's done in the NAS-T codebase is with a [[../../.dir-locals.el][.dir-locals.el]] file.
7.86+
7.87+You just need to add this form for the mode of your choice, replacing the string
7.88+with a regular expression which matches on a /heading/. In this case we treat lines
7.89+starting with three comment chars or more as a new heading.
7.90+#+begin_src lisp-data
7.91+(makefile-mode . ((outline-regexp . "###+")))
7.92+#+end_src
7.93+
7.94+=outline-regexp= is declared as a safe local var, so no prompts will appear asking if
7.95+you trust these values. You will need to configure your keybinds and enable the
7.96+minor-mode separately though. For project-level support, that's all there is to it.
7.97+
7.98+*** Using init.el
7.99+You may also modify your config to enable =outline-minor-mode= for select major-modes at
7.100+startup. Here's a quick example from my config:
7.101+
7.102+#+begin_src emacs-lisp
7.103+;;; Code:
7.104+(require 'default 'rw/fu)
7.105+
7.106+(defun outline-hook (rx)
7.107+ "Enable `outline-minor-mode' and set `outline-regexp'."
7.108+ (setq-local outline-regexp rx)
7.109+ (outline-minor-mode t))
7.110+
7.111+(defun add-outline-hook (mode rx)
7.112+ (let ((sym (symb mode "-hook")))
7.113+ (add-hook sym (lambda () (outline-hook rx)))))
7.114+
7.115+(defmacro outline-hooks (&rest pairs)
7.116+ `(mapc (lambda (x) (add-outline-hook (car x) (cadr x))) ',pairs))
7.117+
7.118+(outline-hooks (asm-mode ";;;+")
7.119+ (nasm-mode ";;;+")
7.120+ (rust-mode "\\(//!\\|////+\\)")
7.121+ (sh-mode "###+")
7.122+ (sh-script-mode "###+")
7.123+ (makefile-mode "###+"))
7.124+
7.125+(provide 'outline-cfg)
7.126+;;; outline-cfg.el ends here
7.127+#+end_src
7.128+** Default Sections
7.129+Our default sections should look familiar - they're just Emacs Lisp defaults, with a few
7.130+choice extensions.
7.131+*** Source Header
7.132+First line of every source code file.
7.133+
7.134+Here is the prototype in lisp:
7.135+#+begin_src lisp
7.136+;;; filename --- description -*- vars -*-
7.137+#+end_src
7.138+
7.139+In Rust we use:
7.140+#+begin_src rust
7.141+//! filename --- description -*- vars -*-
7.142+#+end_src
7.143+
7.144+etc.
7.145+**** Metadata :optional:
7.146+Some files may insert a blank line and start the =Code= heading, while others will
7.147+include some additional information about the file such as a long-description, version,
7.148+list of exports, etc.
7.149+*** Commentary :optional:
7.150+An optional programmer commentary included in source code files after the =Source
7.151+Header= but before the =Code=. The contents are unpredictable but may include notes,
7.152+todos, diagrams, stack notations, test results, links, tips, etc.
7.153+*** Code
7.154+The =Code= heading should be the final toplevel heading of any source code file. You
7.155+may see a number of sub-headings, starting with four or more comment chars.
8.1--- a/hello-world.org Tue Dec 12 21:50:58 2023 -0500
8.2+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
8.3@@ -1,90 +0,0 @@
8.4-{{{header(hello world,
8.5-Richard Westhaver,
8.6-ellis@rwest.io)}}}
8.7-#+options: toc:t h:1
8.8-* Introduction
8.9-Hello World,
8.10-
8.11-I've worked in and around software systems for many years. Throughout
8.12-my journey I've worked with some incredibly capable programmers,
8.13-leaders, sales professionals, and curious minds from all walks of
8.14-life. I've been fortunate to witness some highly effective teams on
8.15-complex projects deliver under impossible constraints. I've also seen
8.16-the other side - where a team isn't effective for one reason or
8.17-another, even on simple problems.
8.18-
8.19-My work as of late is stemmed from the simple premise that there
8.20-exists an /environment/ in which a team can be highly effective. My
8.21-goal is to find those local optimums in my areas of interest where
8.22-such an environment can be built and /great work/ can be done.
8.23-
8.24-In my professional experience, it is becoming much more difficult to
8.25-build an /environment/ where people can be effective. There are
8.26-several contributing factors which muddy the waters but it all boils
8.27-down to capabilities and politics.
8.28-
8.29-Your environment must be capable. It must provide everything your team
8.30-needs now and /may/ need in the future. If the environment doesn't
8.31-provide something, you need to provide all building materials for it -
8.32-/even if you don't know what you're building yet/.
8.33-
8.34-To build a capable environment you need to discard politics from your
8.35-decision-making process. In other words, drop the ego. This requires a
8.36-high degree of introspection. It's hard enough for individuals, let
8.37-alone an entire team or company. In the world of software we tend to
8.38-have two camps - the pro-dev who prefers ergonomic but proprietary
8.39-tools and the foss-dev who prefers clunky but open-source
8.40-alternatives. You can't limit your environment based on the camp you
8.41-align with.
8.42-
8.43-* The Compiler Company
8.44-Without further ado, I'd like to announce /The Compiler Company,
8.45-LLC/. The purpose of /The Compiler Company/ is to /compile/
8.46-/companies/.
8.47-
8.48-More specifically, I'm writing a software suite which specializes in
8.49-building environments.
8.50-
8.51-The software isn't for everyone - modules will be rewritten
8.52-frequently, code may be terse in places, with specialized tools,
8.53-custom compilers, and advanced hardware features. It's for a specific
8.54-type of programmer - an /operator/ if you will, who may use it for
8.55-rapid-development of their own programs (or companies). The barrier to
8.56-entry is high.
8.57-
8.58-At this stage, I'm interested in *systems*, *processes*, and
8.59-*protocols* - not *products* and *services*. /The Compiler Company/'s
8.60-most valuable assett its ideas. A /demo/ system is being written which
8.61-serves as a reference implementation but this is currently designed
8.62-for internal benchmarking.
8.63-
8.64-** [[https://compiler.company/docs/core][core]]
8.65-The =core= is a collection of applications and libraries built from
8.66-the bottom-up with modularity in mind. It's primarily written in
8.67-Common Lisp and Rust with minimal external dependencies.
8.68-
8.69-*Lisp* is a first-class citizen of our internal environment. We
8.70-currently rely on the Steel Bank Common Lisp compiler but even if we
8.71-switch to a different implementation there will always be Lisp. It's
8.72-our dedicated high-level programming language (and much more, as we'll
8.73-explain later).
8.74-
8.75-*Rust* is second-class. It meets an arbitrary criteria for what I
8.76-would consider /good enough/ but there are many contenders including
8.77-C, C++, and Zig. It helps fill the gaps in our Lisp environment which
8.78-would be extremely difficult to implement from scratch like
8.79-eliminating GC, compile-time type safety, cross-compilation features,
8.80-and advanced networking protocols. The community support and my
8.81-personal experience with the language are also contributing
8.82-factors. The trade-off is that we need to support another language
8.83-environment in parallel to Lisp.
8.84-
8.85-** [[https://compiler.company/docs/infra][infra]]
8.86-Unfortunately, ideas can't host themselves. We need a robust
8.87-infrastructure to compensate for this. The project =infra= contains
8.88-scripts for building and maintaing the entire corporate
8.89-infrastructure.
8.90-
8.91-We typically host services on Arch Linux. Podman and QEMU are used for
8.92-virtualization. Modules can be built and deployed separately to make
8.93-host-migration easier as we expand.
9.1--- a/readme.org Tue Dec 12 21:50:58 2023 -0500
9.2+++ b/readme.org Sun Apr 28 19:49:20 2024 -0400
9.3@@ -1,6 +1,29 @@
9.4-{{{header(blog,
9.5-Richard Westhaver,
9.6-ellis@rwest.io,
9.7-The Compiler Company Blog)}}}
9.8+#+title: blog
9.9 #+EXPORT_FILE_NAME: index
9.10-* [2023-11-19 Sun] [[file:hello-world.org][hello-world]]
9.11+* posts
9.12+* drafts
9.13+** DRAFT A Bit of RISC
9.14+:LOGBOOK:
9.15+- State "DRAFT" from [2024-04-17 Wed 18:51]
9.16+:END:
9.17+
9.18+** OUTLINE A Lispy Database
9.19+:LOGBOOK:
9.20+- State "OUTLINE" from "DRAFT" [2024-04-17 Wed 18:53]
9.21+:END:
9.22+
9.23+** DRAFT Shared Library Skeletons
9.24+:LOGBOOK:
9.25+- State "DRAFT" from [2024-04-17 Wed 18:53]
9.26+:END:
9.27+
9.28+** RESEARCH On Lisp Ecosystems
9.29+:LOGBOOK:
9.30+- State "DRAFT" from [2024-04-17 Wed 18:53]
9.31+- State "RESEARCH" from "DRAFT" [2024-04-17 Wed 18:53]
9.32+:END:
9.33+
9.34+** DRAFT Outlines
9.35+:LOGBOOK:
9.36+- State "DRAFT" from [2024-04-17 Wed 18:53]
9.37+:END:
