# HG changeset patch # User Richard Westhaver # Date 1714348160 14400 # Node ID d77884ec2b44578aed4268cd374d559cd7643470 # Parent b6380832df99ead6c390a664bdf3b5623444a978 drafts diff -r b6380832df99 -r d77884ec2b44 .hgignore --- a/.hgignore Tue Dec 12 21:50:58 2023 -0500 +++ b/.hgignore Sun Apr 28 19:49:20 2024 -0400 @@ -0,0 +1,1 @@ +[.]html \ No newline at end of file diff -r b6380832df99 -r d77884ec2b44 draft/a-bit-of-risc.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/draft/a-bit-of-risc.org Sun Apr 28 19:49:20 2024 -0400 @@ -0,0 +1,310 @@ +#+title: A Bit of RISC +#+date: [2024-03-11 Mon] +I recently picked up [[https://dl.acm.org/doi/10.5555/2462741][Hacker's Delight]] and having a lot of fun with +it. It's a collection of bit-manipulation tricks collected by hackers +over many years. You can flip open pretty much anywhere in the book +and start learn something really cool. + +There's something about seeing bit strings and assembly code in a book +that really catches my attention, this one goes a bit further by even +describing a complete RISC (Reduced Instruction Set Computer) we can +implement to play around with the various tricks. + +As an exercise and for fun, I'd like to employ some Lisp-fu here and +implement a small VM specifically designed for mangling bits. + +As a fair warning, I'm not a mathematician and I don't write proofs +often. If I get something wrong or there is a better way of doing +things, let me know! + +You can find most of the code from the book [[https://github.com/hcs0/Hackers-Delight][here]]. + +* Design +** Data Representation +We'll be sticking with a 32-bit word length as recommended in the +Preface. We will also represent registers as integers whenever +possible, instead of say a bit-vector. Without going into too much +detail, it's much more efficient to do bitwise ops on integers +instead of bit-vectors in Lisp. + +We need a minimum of 16 general purpose registers, typically of word +length, with R0 reserved for a constant 0. To address 16 different +registers we actually only need 4 bits - 5-bits if we needed 32, 6 for +64, etc. + +Floating-point support and special purpose registers are not required. + +** Instructions +The Hacker's Delight RISC architecture is described in two tables, +denoted =basic RISC= and =full RISC= respectively. + +Most instructions take two source registers =RA= and =RB= with a +destination register =RT=. The actual general-purpose registers are +labelled =R0= (containg the constant 0) through =R15=. + +A 3-Address machine is assumed and some instructions take 16-bit +signed or unsigned immediate values - denoted =I= and =Iu= +respectively. + +#+tblname: Basic Instruction Set (basic RISC) +| Opcode Mnemonic | Operands | Description | +|-----------------------------------------------------------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------| +| add,sub,mul,div,divu,rem,remu | RT,RA,RB | RT <- (op RA RB) | +| addi,muli | RT,RA,I | RT <- (op RA I), I is a 16-bit signed immediate-value | +| addis | RT,RA,I | RT <- (+ RA (\vert\vert I 0x0000)) | +| and,or,xor | RT,RA,RB | RT <- (op RA RB) | +| andi,ori,xori | RT,RA,Iu | As above, expect the last operand is a 16-bit unsigned immediate-value | +| beq,bne,blt,ble,bgt,bge | RT,target | Branch to target if (op RT) | +| bt,bf | RT,target | Branch true/false, same as bne/beq resp | +| cmpeq,cmpne,cmplt,cmple,cmpgt,cmpge,cmpltu,cmpleu,cmpgtu,cmpgeu | RT,RA,RB | RT <- (if (op RA RB) 1 0) | +| cmpieq,cmpine,cmpilt,cmpile,cmpigt,cmpige | RT,RA,I | Like cmpeq except second comparand is a 16-bit signed immediate-value | +| cmpiequ,cmpineu,cmpiltu,cmpileu,cmpigtu,cmpigeu | RT,RA,I | Like cmpltu except second comparand is a 16-bit unsigned immediate-value | +| ldbu,ldh,ldhu,ldw | RT,d(RA) | Load an unsigned-byte, signed-halfword, unsigned-halfword, or word into RT from (+ RA d) where d is a 16-bit signed immediate-value | +| mulhs,mulhu | RT,RA,RB | RT gets the high-order 32 bits of (* RA RB) | +| not | RT,RA | RT <- bitwise one's-complement of RA | +| shl,shr,shrs | RT,RA,RB | RT <- RA shifted left or right by rightmost six bits of RB; 0-fill except for shrs, which is sign-fill (shift amount treated modulo 64) | +| shli,shri,shrsi | RT,RA,Iu | RT <- RA shifted left or right by 5-bit immediate field | +| stb,sth,stw | RS,d(RA) | Store a byte,halfword,word from RS into memory at location (+ RA d) where d is a 16-bit signed immediate-value | + + +#+name: Addition Instructions (full RISC) +| Opcode Mnemonic | Operands | Description | +|-----------------------------------------------------------------+-----------+--------------------------------------------------------------------------------------------------------| +| abs,nabs | RT,RA | RT <- (op RA) | +| andc,eqv,nand,nor,orc | RT,RA,RB | RT <- (op RA RB) | +| extr | RT,RA,I,L | extract bits I through I+L-1 of RA and place them right-adjusted in RT, with 0-fill | +| extrs | RT,RA,I,L | Like extr, but sign-fill | +| ins | RT,RA,I,L | Insert bits 0 through L-1 of RA into bits I through I+L-1 of RT | +| nlz | RT,RA | RT gets count of leading 0's in RA (0 to 32) | +| pop | RT,RA | RT gets the number of 1-bits in RA (0 to 32) | +| ldb | RT,d(RA) | Load a signed byte into RT from memory at location (+ RA d) where d is a 16-bit signed immediate value | +| moveq,movne,movlt,movle,movgt,movge | RT,RA,RB | RT <- RA rotate-shifted left or right by the rightmost 5-bits of RB | +| shlr,shrr | RT,RA,RB | RT <- RA rotate-shifted left or right by the rightmost 5-bits of RB | +| shlri,shrri | RT,RA,Iu | RT <- RA rotate-shifted left or right by the 5-bit immediate field | +| trpeq,trpne,trplt,trple,trpgt,trpge,trpltu,trpleu,trpgtu,trpgeu | RA,RB | Trap (interrupt) if (op RA RB) | +| trpieq,trpine,trpilt,trpile,trpigt,trpige | RA,I | Trap if (op RA I) where I is a 16-bit signed immediate-value | +| trpiequ,trpineu,trpiltu,trpileu,trpigtu,trpigeu | RA,Iu | Trap if (op RA Iu) where Iu is a 16-bit unsigned immediate-value | + +There is also some extensions, which are like macros that usually +expand to a single instruction. + +#+name: Extended Mnemonics +| Extended Mnemonic | Expansion | Description | +|-------------------+------------------+---------------------------| +| b target | beq R0,target | Unconditional branch | +| li RT,I | (addi,addis,ori) | Load immediate | +| mov RT,RA | ori RT,RA,0 | Move register RA to RT | +| neg RT,RA | sub RT,R0,RA | Negate (two's-complement) | +| subi RT,RA,I | addi RT,RA,-I | Subtract immediate | + +All of these instructions are available on x86,arm,riscv and the likes +so no real surprises. We will implement the basic set in Lisp, mapping +instructions directly to Lisp functions using macros. + +** Execution Model +We'll build this machine in Lisp and use plenty intrinsics from +SBCL. As a starting point I followed Paul Khuong's excellent blog +post: [[https://pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembly-code-breadboard/][SBCL: The ultimate assembly code breadboard]]. + +Some things to keep in mind for our machine: +- every instruction requires at most two register reads and one + register write - good for compilers +- every instruction counts as a single cycle +- we pay no attention to instruction-level parallelism + +* The HAKMEM VM +:properties: +:header-args: :session t :results none +:end: +#+name: defpackage +#+begin_src lisp + (ql:quickload :prelude) + (in-package :std-user) + (defpackage :hakmem + (:use :cl :std) + (:import-from :sb-assem :inst) + (:import-from :sb-vm :immediate-constant :registers :zero :ea)) + (in-package :hakmem) + ;; (in-package :sb-x86-64-asm) + (in-readtable :std) + (declaim (optimize (speed 3) (safety 1))) + (defconstant +word-size+ 32 "default word size and register length.") +#+end_src + +#+name: vars +#+begin_src lisp :package hakmem + (declaim (type (unsigned-byte #.+word-size+) +ro+)) + (defconstant +r0+ 0 "constant value for register 0") + (defvar *stack* (make-array 8 :initial-contents (list sb-vm::r8-tn + sb-vm::r9-tn + sb-vm::r10-tn + sb-vm::r11-tn + sb-vm::r12-tn + sb-vm::r13-tn + sb-vm::r14-tn + sb-vm::r15-tn))) + (defvar *stack-pointer*) + + (defvar *rax* sb-vm::rax-tn) + (defvar *rbx* sb-vm::rax-tn) + (defvar *rcx* sb-vm::rax-tn) + (defvar *rdx* sb-vm::rax-tn) + + ;; (@ 0) returns the (current) register for TOS, (@ 1) returns + ;; the one just below, etc. + (defun @ (i) + (aref *stack* (mod (+ i *stack-pointer*) (length *stack*)))) + + (defvar *code-base* sb-vm::rsi-tn) + (defvar *virtual-ip* sb-vm::rdi-tn) + (sb-x86-64-asm::get-gpr :qword 4) + ;; (sb-vm::immediate-constant-sc 10000) + ;; arena vector or list? + (defvar *instructions* (make-hash-table :test #'equal)) + + (defvar *primitive-code-offset* (* 64 67)) + + (defstruct code-page + (alloc 0) ;; next free byte + (code (make-array *primitive-code-offset* :element-type 'octet))) + + (defun emit-code (pages emitter) + ;; there must be as many code pages as there are stack slots + (assert (= (length *stack*) (length pages))) + ;; find the rightmost starting point, and align to 16 bytes + (let* ((alloc (logandc2 (+ 15 (reduce #'max pages :key #'code-page-alloc)) + 15)) + (bytes (loop for i below (length pages) + for page = (elt pages i) + collect (let ((segment (sb-assem:make-segment)) + (*stack-pointer* i)) + ;; assemble the variant for this value + ;; of *stack-pointer* in a fresh code + ;; segment + (sb-assem:assemble (segment) + ;; but first, insert padding + (sb-vm::emit-long-nop segment (- alloc (code-page-alloc page))) + (funcall emitter)) + ;; tidy up any backreference + (sb-assem:finalize-segment segment) + ;; then get the (position-independent) machine + ;; code as a vector of bytes + (sb-assem:segment-contents-as-vector segment))))) + ;; finally, copy each machine code sequence to the right code page + (map nil (lambda (page bytes) + (let ((alloc (code-page-alloc page))) + (replace (code-page-code page) bytes :start1 alloc) + (assert (<= (+ alloc (length bytes)) (length (code-page-code page)))) + (setf (code-page-alloc page) (+ alloc (length bytes))))) + pages bytes) + ;; and return the offset for that code sequence + alloc)) + + (defun emit-all-code (&rest emitters) + (let ((pages (loop repeat (length *stack*) + for page = (make-code-page) + ;; prefill everything with one-byte NOPs + do (fill (code-page-code page) #x90) + collect page))) + (values (mapcar (lambda (emitter) + (emit-code pages emitter)) + emitters) + pages))) + + (defun next (&optional offset) + (setf offset (or offset 0)) ; accommodate primops that frob IP + (let ((rotation (mod *stack-pointer* (length *stack*)))) + (inst movzx *rax* (make-ea :dword :base *virtual-ip* + :disp offset)) + (unless (= -4 offset) + (inst add *virtual-ip* (+ 4 offset))) + (if (zerop rotation) + (inst add *rax* *code-base*) + (inst lea *rax* (make-ea :qword :base *code-base* + :index *rax* + :disp (* rotation *primitive-code-offset*)))) + (inst jmp *rax*))) + + (defun swap () + (inst xchg (@ 0) (@ 1)) ; exchange top of stack and stack[1] + (next)) + +#+end_src + +#+name: instructions +#+begin_src lisp :package hakmem + ;; todo + (defun %parse-reg3 (rt ra rb)) + (defun %parse-reg2i (rt ra i)) + (defun %parse-reg2ui (rt ra ui)) + (defmacro def-inst (name args &body body) + ;; todo: compose a function based on regs+args+body + `(let ((sc *scratch*) + (r0 +r0+) + (ra 0) + (rb 0) + (rt 0)) + (declare (ignorable sc r0 ra rb rt)) + (setf (gethash ',name *instructions*) (lambda ,args (progn ,@body))))) + + (defmacro def-prim (name op) + `(def-inst ,name () (setf rt (,op ra rb)))) +#+end_src + +#+name: prims +#+begin_src lisp :package hakmem + (def-prim add +) + (def-prim sub -) + (def-prim mul *) + (def-prim div /) + ;; divu + (def-prim rem mod) + ;; remu + (def-prim cmpeq =) + (def-prim cmpne /=) + (def-prim cmplt <) + (def-prim cmple <=) + (def-prim cmpgt >) + (def-prim cmpge >=) + ;; ltu leu gtu geu + (def-inst addi (i) + (setf rt (+ ra i))) + (def-inst muli (i) + (setf rt (* ra i))) + (def-prim and logand) + (def-prim or logior) + (def-prim xor logxor) + + (defun get-inst (i) (gethash i *instructions*)) + + (defmacro %inst (i &body args) + `(funcall (get-inst ',i) ,@args)) + + (defun list-instructions (&optional (tbl *instructions*)) + (hash-table-alist tbl)) + +#+end_src +#+name: instruction-list +#+begin_src lisp :results replace :exports both :package hakmem + (list-instructions) +#+end_src + +#+RESULTS: instruction-list +#+begin_example +((XOR . #) + (OR . #) + (AND . #) + (MULI . #) + (ADDI . #) + (CMPGE . #) + (CMPGT . #) + (CMPLE . #) + (CMPLT . #) + (CMPNE . #) + (CMPEQ . #) + (REM . #) + (DIV . #) + (MUL . #) + (SUB . #) + (ADD . #)) +#+end_example diff -r b6380832df99 -r d77884ec2b44 draft/a-lispy-database.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/draft/a-lispy-database.org Sun Apr 28 19:49:20 2024 -0400 @@ -0,0 +1,25 @@ +#+title: A Lispy Database +#+options: toc:t h:1 +One of the key features missing in the =core= is a DBMS. The first, +and often only choice for this need in companies of today is +a SQL RDBMS. + +There are client-server systems like PostgreSQL and MySQL, and +embedded offerings like SQLite. Whatever lang you use you can count on +there being SQL support[fn:1]. To support all the different SQL +flavors though, a new abstraction has been summoned - the ORM. + +The ORM maps the object system of your lang to SQL tables, columns and +rows and provides an API for you to manipulate the database in a more +idiomatic way. + +You might ask yourself, why choose anything else? For good reason. We +all know friendly neighborhood SQL and understand it. Many of us have +never used anything else because we don't /need/ to. You can represent +fairly complex relationships in the RDBMS model and have an +ecosystem that supports it wherever you go. + +My answer is that just because SQL is good enough, doesn't mean it's +always the best choice. + +[fn:1] diff -r b6380832df99 -r d77884ec2b44 draft/dylib-skel.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/draft/dylib-skel.org Sun Apr 28 19:49:20 2024 -0400 @@ -0,0 +1,146 @@ +#+title: Shared Library Skeletons +* Overview ++ CODE :: [[https://lab.rwest.io/packy/stash/dysk][packy/stash/dysk]] +Our core languages are [[https://www.rust-lang.org/][Rust]] and [[https://lisp-lang.org/][Lisp]] - this is the killer combo which will allow NAS-T +to rapidly develop high-quality software. As such, it's crucial that these two very +different languages (i.e. compilers) are able to interoperate seamlessly. + +Some interop methods are easy to accomodate via the OS - such as IPC or data sharing, +but others are a bit more difficult. + +In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something +that /can/ be difficult, due to some complications with Rust and because this is not the +most popular software stack (yet ;). This is an experiment and may not make it to our +code-base, but it's definitely something worth adding to the toolbox in case we need it. +** FFI +The level of interop we're after in this case is [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFI]]. + +Basically, calling Rust code from Lisp and vice-versa. There's an article about calling +Rust from Common Lisp [[https://dev.to/veer66/calling-rust-from-common-lisp-45c5][here]] which shows the basics and serves as a great starting point +for those interested. +*** Rust ABI +The complication(s) with Rust I mentioned early is really just that /it is not C/. =C= +is old, i.e. well-supported with a stable ABI, making the process of creating bindings +for a C library a breeze in many languages. + +For a Rust library we need to first appease the compiler, as explained in [[https://doc.rust-lang.org/nomicon/ffi.html#calling-rust-code-from-c][this section]] +of the Rustonomicon. Among other things it involves changing the calling-convention of +functions with a type signature and editing the Cargo.toml file to produce a +C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used +like the C ABI can. + +[[https://github.com/rodrimati1992/abi_stable_crates][abi_stable_crates]] is a project which addresses some of the ABI concerns, presenting a +sort of ABI-API as a Rust library. Perhaps this is the direction the ecosystem will go +with in order to maintain an unstable ABI, but for now there is no 'clear' pathway for a +friction-less FFI development experience in Rust. + +*** Overhead +Using FFI involves some overhead. Check [[https://github.com/dyu/ffi-overhead][here]] for an example benchmark across a few +languages. While building the NAS-T core, I'm very much aware of this, and will need a +few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular, +I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp). + +* basic example +** Setup +For starters, I'm going to assume we all have Rust (via [[https://rustup.rs/][rustup]]) and Lisp ([[https://www.sbcl.org/][sbcl]] only) +installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in +this post). +*** Cargo +Create a new library crate. For this example we're focusing on a 'skeleton' for +/dynamic/ libraries only, so our experiment will be called =dylib-skel= or *dysk* for +short. +src_sh[:exports code]{cargo init dysk --lib && cd dysk} + +A =src/lib.rs= will be generated for you. Go ahead and delete that. We're going to be +making our own =lib.rs= file directly in the root directory (just to be cool). + +The next step is to edit your =Cargo.toml= file. Add these lines after the =[package]= +section and before =[dependencies]=: +#+begin_src conf-toml +[lib] +crate-type = ["cdylib","rlib"] +path = "lib.rs" +[[bin]] +name="dysk-test" +path="test.rs" +#+end_src + +This tells Rust to generate a shared C-compatible object with a =.so= extension which we +can open using [[https://man.archlinux.org/man/dlopen.3.en][dlopen]]. +*** cbindgen +**** install +Next, we want the =cbindgen= program which we'll use to generate header files for +C/C++. This step isn't necessary at all, we just want it for further experimentation. + +src_sh[:exports code]{cargo install --force cbindgen} + +We append the =cbindgen= crate as a /build dependency/ to our =Cargo.toml= like so: +#+begin_src conf-toml +[build-dependencies] +cbindgen = "0.24" +#+end_src +**** cbindgen.toml +#+begin_src conf-toml :tangle cbindgen.toml +language = "C" +autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */" +include_version = true +namespace = "dysk" +cpp_compat = true +after_includes = "#define DYSK_VERSION \"0.1.0\"" +line_length = 88 +tab_width = 2 +documentation = true +documentation_style = "c99" +usize_is_size_t = true +[cython] +header = "dysk.h" +#+end_src +**** build.rs +#+begin_src rust :tangle build.rs +fn main() -> Result<(), cbindgen::Error> { + if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) { + b.write_to_file("dysk.h"); Ok(())} + else { panic!("failed to generate dysk.h from cbindgen.toml") } } +#+end_src +** lib.rs +#+begin_src rust :tangle lib.rs +//! lib.rs --- dysk library +use std::ffi::{c_char, c_int, CString}; +#[no_mangle] +pub extern "C" fn hello() -> *const c_char { + CString::new("hello from rust").unwrap().into_raw()} +#[no_mangle] +pub extern "C" fn plus(a:c_int,b:c_int) -> c_int {a+b} +#[no_mangle] +pub extern "C" fn plus1(n:c_int) -> c_int {n+1} +#+end_src +** test.rs +#+begin_src rust :tangle test.rs +//! test.rs --- dysk test +fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::plus1(2 as core::ffi::c_int);}} +#+end_src +** compile +#+begin_src sh +cargo build --release +#+end_src +** load from SBCL +#+begin_src lisp :tangle dysk.lisp +;;; dysk.lisp +;; (dysk:hello) ;; => "hello from rust" +(defpackage :dysk + (:use :cl :sb-alien) + (:export :hello :plus :plus1)) +(in-package :dysk) +(load-shared-object #P"target/release/libdysk.so") +(define-alien-routine hello c-string) +(define-alien-routine plus int (a int) (b int)) +(define-alien-routine plus1 int (n int)) +#+end_src +** benchmark +#+begin_src shell +time target/release/dysk-test +#+end_src +#+begin_src lisp :tangle test.lisp +(time (dotimes (_ 500000000) (dysk:plus1 2))) +#+end_src + diff -r b6380832df99 -r d77884ec2b44 draft/hello-world.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/draft/hello-world.org Sun Apr 28 19:49:20 2024 -0400 @@ -0,0 +1,88 @@ +#+title: (hello world) +#+options: toc:t h:1 +* COMMENT Introduction +Hello World, + +I've worked in and around software systems for many years. Throughout +my journey I've worked with some incredibly capable programmers, +leaders, sales professionals, and curious minds from all walks of +life. I've been fortunate to witness some highly effective teams on +complex projects deliver under impossible constraints. I've also seen +the other side - where a team isn't effective for one reason or +another, even on simple problems. + +My work as of late is stemmed from the simple premise that there +exists an /environment/ in which a team can be highly effective. My +goal is to find those local optimums in my areas of interest where +such an environment can be built and /great work/ can be done. + +In my professional experience, it is becoming much more difficult to +build an /environment/ where people can be effective. There are +several contributing factors which muddy the waters but it all boils +down to capabilities and politics. + +Your environment must be capable. It must provide everything your team +needs now and /may/ need in the future. If the environment doesn't +provide something, you need to provide all building materials for it - +/even if you don't know what you're building yet/. + +To build a capable environment you need to discard politics from your +decision-making process. In other words, drop the ego. This requires a +high degree of introspection. It's hard enough for individuals, let +alone an entire team or company. In the world of software we tend to +have two camps - the pro-dev who prefers ergonomic but proprietary +tools and the foss-dev who prefers clunky but open-source +alternatives. You can't limit your environment based on the camp you +align with. + +* COMMENT The Compiler Company +Without further ado, I'd like to announce /The Compiler Company, +LLC/. The purpose of /The Compiler Company/ is to /compile/ +/companies/. + +More specifically, I'm writing a software suite which specializes in +building environments. + +The software isn't for everyone - modules will be rewritten +frequently, code may be terse in places, with specialized tools, +custom compilers, and advanced hardware features. It's for a specific +type of programmer - an /operator/ if you will, who may use it for +rapid-development of their own programs (or companies). The barrier to +entry is high. + +At this stage, I'm interested in *systems*, *processes*, and +*protocols* - not *products* and *services*. /The Compiler Company/'s +most valuable assett its ideas. A /demo/ system is being written which +serves as a reference implementation but this is currently designed +for internal benchmarking. + +** [[https://compiler.company/docs/core][core]] +The =core= is a collection of applications and libraries built from +the bottom-up with modularity in mind. It's primarily written in +Common Lisp and Rust with minimal external dependencies. + +*Lisp* is a first-class citizen of our internal environment. We +currently rely on the Steel Bank Common Lisp compiler but even if we +switch to a different implementation there will always be Lisp. It's +our dedicated high-level programming language (and much more, as we'll +explain later). + +*Rust* is second-class. It meets an arbitrary criteria for what I +would consider /good enough/ but there are many contenders including +C, C++, and Zig. It helps fill the gaps in our Lisp environment which +would be extremely difficult to implement from scratch like +eliminating GC, compile-time type safety, cross-compilation features, +and advanced networking protocols. The community support and my +personal experience with the language are also contributing +factors. The trade-off is that we need to support another language +environment in parallel to Lisp. + +** [[https://compiler.company/docs/infra][infra]] +Unfortunately, ideas can't host themselves. We need a robust +infrastructure to compensate for this. The project =infra= contains +scripts for building and maintaing the entire corporate +infrastructure. + +We typically host services on Arch Linux. Podman and QEMU are used for +virtualization. Modules can be built and deployed separately to make +host-migration easier as we expand. diff -r b6380832df99 -r d77884ec2b44 draft/on-lisp-ecosystems.org diff -r b6380832df99 -r d77884ec2b44 draft/outlines.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/draft/outlines.org Sun Apr 28 19:49:20 2024 -0400 @@ -0,0 +1,152 @@ +#+title: outlines +* Overview +Source code files are hard to manage. They can get unwieldly quickly and making the +wrong assumption about your whereabouts in the code tree can have unintended +consequences. + +There are many ways to solve this problem to different degrees. We'll be talking about +one strategy in particular which I use and recommend for any software project. + +Looking through the source code in the NAS-T repository you'll find some common +commenting patterns: + +- every file start with at least one comment line for example: +#+begin_src lisp +;;; file-name.lisp --- file description +#+end_src + +- Before you see any code in a file, you'll likely encounter this line: +#+begin_src lisp +;;; Code: +#+end_src + +- etc + +What's the deal here? To be clear, I'm of the mind that comments should be +significant. They should express to the reader something that is of a non-trivial nature +and 'where the code starts' doesn't quite qualify. Indeed, these comments don't fit that +model at all. + +The deal is that these comments aren't for the reader, they're for the developer. More +specifically, for the developer to treat as a special meta-language to describe the +structure of a source code file. + +* Outlines +Like all my good ideas, this one is credited entirely to Emacs. In this case, the +excellent [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Outline-Mode.html][Outline mode]]. If you are an Emacs user you've probably already used it without +knowing -- Org mode, for example, is [[https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/org.el?h=release_9.6.9#n4789][derived from outline-mode]]. + +I've grown quite fond of it. Here's the summary: + +#+begin_quote +Outline mode is a major mode derived from Text mode, which is specialized for editing +outlines. It provides commands to navigate between entries in the outline structure, and +commands to make parts of a buffer temporarily invisible, so that the outline structure +may be more easily viewed. +#+end_quote +-- [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Outline-Mode.html][GNU]] + +** Quickstart +If you want to jump in right away, I recommend using these keybinds in Emacs: + +#+tblname: outline-keys +| | outline-cycle-buffer | +| M-TAB | outline-cycle | +| M-n | outline-next-visible-heading | +| M-p | outline-previous-visible-heading | + +Here's a snippet which will enable the keybinds I use: + +#+name: enable-outline-keys +#+begin_src emacs-lisp +(let ((keys + '(("" #'outline-cycle-buffer) + ("M-TAB" #'outline-cycle) + ("M-n" #'outline-next-visible-heading) + ("M-p" #'outline-previous-visible-heading)))) + (cl-loop for (k fn) in keys + do (keymap-set outline-minor-mode-map k fn))) +#+end_src + +Now open a file in the [[../../src/][src]] directory, like [[../../src/fs/btrfs/btrfs.lisp][this]] one, enable =outline-minor-mode= and +move around the file with the new keybinds above. + +** Outlines4All +Not all programming modes have outline support built-in. The good news is that it's easy +to enable it. + +You only need to modify one variable: =outline-regexp= and enable a minor-mode: +=outline-minor-mode=. + +*** Using dir-locals +The way it's done in the NAS-T codebase is with a [[../../.dir-locals.el][.dir-locals.el]] file. + +You just need to add this form for the mode of your choice, replacing the string +with a regular expression which matches on a /heading/. In this case we treat lines +starting with three comment chars or more as a new heading. +#+begin_src lisp-data +(makefile-mode . ((outline-regexp . "###+"))) +#+end_src + +=outline-regexp= is declared as a safe local var, so no prompts will appear asking if +you trust these values. You will need to configure your keybinds and enable the +minor-mode separately though. For project-level support, that's all there is to it. + +*** Using init.el +You may also modify your config to enable =outline-minor-mode= for select major-modes at +startup. Here's a quick example from my config: + +#+begin_src emacs-lisp +;;; Code: +(require 'default 'rw/fu) + +(defun outline-hook (rx) + "Enable `outline-minor-mode' and set `outline-regexp'." + (setq-local outline-regexp rx) + (outline-minor-mode t)) + +(defun add-outline-hook (mode rx) + (let ((sym (symb mode "-hook"))) + (add-hook sym (lambda () (outline-hook rx))))) + +(defmacro outline-hooks (&rest pairs) + `(mapc (lambda (x) (add-outline-hook (car x) (cadr x))) ',pairs)) + +(outline-hooks (asm-mode ";;;+") + (nasm-mode ";;;+") + (rust-mode "\\(//!\\|////+\\)") + (sh-mode "###+") + (sh-script-mode "###+") + (makefile-mode "###+")) + +(provide 'outline-cfg) +;;; outline-cfg.el ends here +#+end_src +** Default Sections +Our default sections should look familiar - they're just Emacs Lisp defaults, with a few +choice extensions. +*** Source Header +First line of every source code file. + +Here is the prototype in lisp: +#+begin_src lisp +;;; filename --- description -*- vars -*- +#+end_src + +In Rust we use: +#+begin_src rust +//! filename --- description -*- vars -*- +#+end_src + +etc. +**** Metadata :optional: +Some files may insert a blank line and start the =Code= heading, while others will +include some additional information about the file such as a long-description, version, +list of exports, etc. +*** Commentary :optional: +An optional programmer commentary included in source code files after the =Source +Header= but before the =Code=. The contents are unpredictable but may include notes, +todos, diagrams, stack notations, test results, links, tips, etc. +*** Code +The =Code= heading should be the final toplevel heading of any source code file. You +may see a number of sub-headings, starting with four or more comment chars. diff -r b6380832df99 -r d77884ec2b44 hello-world.org --- a/hello-world.org Tue Dec 12 21:50:58 2023 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,90 +0,0 @@ -{{{header(hello world, -Richard Westhaver, -ellis@rwest.io)}}} -#+options: toc:t h:1 -* Introduction -Hello World, - -I've worked in and around software systems for many years. Throughout -my journey I've worked with some incredibly capable programmers, -leaders, sales professionals, and curious minds from all walks of -life. I've been fortunate to witness some highly effective teams on -complex projects deliver under impossible constraints. I've also seen -the other side - where a team isn't effective for one reason or -another, even on simple problems. - -My work as of late is stemmed from the simple premise that there -exists an /environment/ in which a team can be highly effective. My -goal is to find those local optimums in my areas of interest where -such an environment can be built and /great work/ can be done. - -In my professional experience, it is becoming much more difficult to -build an /environment/ where people can be effective. There are -several contributing factors which muddy the waters but it all boils -down to capabilities and politics. - -Your environment must be capable. It must provide everything your team -needs now and /may/ need in the future. If the environment doesn't -provide something, you need to provide all building materials for it - -/even if you don't know what you're building yet/. - -To build a capable environment you need to discard politics from your -decision-making process. In other words, drop the ego. This requires a -high degree of introspection. It's hard enough for individuals, let -alone an entire team or company. In the world of software we tend to -have two camps - the pro-dev who prefers ergonomic but proprietary -tools and the foss-dev who prefers clunky but open-source -alternatives. You can't limit your environment based on the camp you -align with. - -* The Compiler Company -Without further ado, I'd like to announce /The Compiler Company, -LLC/. The purpose of /The Compiler Company/ is to /compile/ -/companies/. - -More specifically, I'm writing a software suite which specializes in -building environments. - -The software isn't for everyone - modules will be rewritten -frequently, code may be terse in places, with specialized tools, -custom compilers, and advanced hardware features. It's for a specific -type of programmer - an /operator/ if you will, who may use it for -rapid-development of their own programs (or companies). The barrier to -entry is high. - -At this stage, I'm interested in *systems*, *processes*, and -*protocols* - not *products* and *services*. /The Compiler Company/'s -most valuable assett its ideas. A /demo/ system is being written which -serves as a reference implementation but this is currently designed -for internal benchmarking. - -** [[https://compiler.company/docs/core][core]] -The =core= is a collection of applications and libraries built from -the bottom-up with modularity in mind. It's primarily written in -Common Lisp and Rust with minimal external dependencies. - -*Lisp* is a first-class citizen of our internal environment. We -currently rely on the Steel Bank Common Lisp compiler but even if we -switch to a different implementation there will always be Lisp. It's -our dedicated high-level programming language (and much more, as we'll -explain later). - -*Rust* is second-class. It meets an arbitrary criteria for what I -would consider /good enough/ but there are many contenders including -C, C++, and Zig. It helps fill the gaps in our Lisp environment which -would be extremely difficult to implement from scratch like -eliminating GC, compile-time type safety, cross-compilation features, -and advanced networking protocols. The community support and my -personal experience with the language are also contributing -factors. The trade-off is that we need to support another language -environment in parallel to Lisp. - -** [[https://compiler.company/docs/infra][infra]] -Unfortunately, ideas can't host themselves. We need a robust -infrastructure to compensate for this. The project =infra= contains -scripts for building and maintaing the entire corporate -infrastructure. - -We typically host services on Arch Linux. Podman and QEMU are used for -virtualization. Modules can be built and deployed separately to make -host-migration easier as we expand. diff -r b6380832df99 -r d77884ec2b44 readme.org --- a/readme.org Tue Dec 12 21:50:58 2023 -0500 +++ b/readme.org Sun Apr 28 19:49:20 2024 -0400 @@ -1,6 +1,29 @@ -{{{header(blog, -Richard Westhaver, -ellis@rwest.io, -The Compiler Company Blog)}}} +#+title: blog #+EXPORT_FILE_NAME: index -* [2023-11-19 Sun] [[file:hello-world.org][hello-world]] +* posts +* drafts +** DRAFT A Bit of RISC +:LOGBOOK: +- State "DRAFT" from [2024-04-17 Wed 18:51] +:END: + +** OUTLINE A Lispy Database +:LOGBOOK: +- State "OUTLINE" from "DRAFT" [2024-04-17 Wed 18:53] +:END: + +** DRAFT Shared Library Skeletons +:LOGBOOK: +- State "DRAFT" from [2024-04-17 Wed 18:53] +:END: + +** RESEARCH On Lisp Ecosystems +:LOGBOOK: +- State "DRAFT" from [2024-04-17 Wed 18:53] +- State "RESEARCH" from "DRAFT" [2024-04-17 Wed 18:53] +:END: + +** DRAFT Outlines +:LOGBOOK: +- State "DRAFT" from [2024-04-17 Wed 18:53] +:END: