# HG changeset patch # User Richard Westhaver # Date 1725483533 14400 # Node ID 45ac54093c0925fde93384addb414d9d654ba706 # Parent a04ca5a66178e77d78d645a4ec95f807e303b530 init log diff -r a04ca5a66178 -r 45ac54093c09 20230730.org --- a/20230730.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,42 +0,0 @@ -* VC infrastructure -:PROPERTIES: -:ID: 2d40eec1-62f4-4f3d-9be7-110400808e06 -:END: -In heptapod we have a root group named =comp=, containg a variety of -subgroups. Some of these groups should be public, while others are -internal to comp members exclusively. Within each subgroup, we should -have the root group members automatically granted privileged access to -projects. This is relevant for the =startup= subgroup in particular, -where each project is potentially maintained by multiple non-root -contributors. - -We also need to consider how we will manage subrepos across the -organization. It is about time we start integrating HG bundles and -potentially mirrors. For our core VC pipeline we should have no -reliance on Git, but this may be difficult. It depends on the behavior -of HG bundles. - -Bookmarks/tags should be used for milestones in the root group and are -infrequent. They are more frequent in projects with a regular release -life-cycle. -* Approaching Webapps -:PROPERTIES: -:ID: 0a52be58-990a-47da-ba5a-a7da7192361c -:END: -I started poking around in the webapp space again so that I can launch -a landing page for NAS-T quickly. The Rust situation has improved -somewhat on the frontend side, and the axum backend stack is nice. - -This might seem like a lot of Rust and not a lot of Lisp, which it is, -but there's still room for Lisp wherever we need it. It mostly plays a -role in the backend, servicing the database and responding to requests -from the Rust edges. All of the important tests for the web APIs are -also written in Lisp. We will almost certainly use Lisp for all static -processing and HTML generation at compile-time. - -This I believe, is the appropriate way to integrate Lisp into a -cutting-edge web-app. You get the good parts of Lisp where you need -them (interactive debugging, dynamic language, REPL) and avoid the bad -parts (OOB optimization, RPS performance) in areas where the customer -would be impacted. In this domain, Lisp takes the form of a glue -rather than the bricks and mortar it sometimes appears to us as. diff -r a04ca5a66178 -r 45ac54093c09 20231024.org --- a/20231024.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,128 +0,0 @@ -* virt -:PROPERTIES: -:ID: a51b54d9-5663-411e-b0ba-d1405622eb88 -:END: -** QEMU -:PROPERTIES: -:ID: 34dc9d12-d651-4204-9abb-a6a9a6aeb07e -:END: -** KVM -:PROPERTIES: -:ID: 909e1127-14f5-4783-97af-65aa25439226 -:END: -** Hyper-V -:PROPERTIES: -:ID: d89dc59d-25d9-4304-bcb8-379dd43cc3f7 -:END: -** Firecracker -:PROPERTIES: -:ID: 7c4f08b7-2bc0-4a85-a6ca-2decdce8dc91 -:END: -** Docker -:PROPERTIES: -:ID: 42ff77c2-69c9-4f89-a297-88efd592502e -:END: -** Vagrant -:PROPERTIES: -:ID: c53c4d87-b0fd-40de-8db7-b92c22dc946c -:END: -** LXC -:PROPERTIES: -:ID: 85c5f471-cf87-49b7-a6ce-5c1526df4080 -:END: -** LXD -:PROPERTIES: -:ID: f92d34dd-7b24-498c-a5c8-29fd3b528092 -:END: -** containerd -:PROPERTIES: -:ID: a967645f-5df3-4bed-9e7b-a1adaed3796c -:END: -** systemd-nspawn -:PROPERTIES: -:ID: 6329c6f9-c54c-4eb0-99e7-4772399161e7 -:END: -** VirtualBox -:PROPERTIES: -:ID: 638aa81a-8926-480a-854f-0434e8119426 -:END: - -* Concatenative -:PROPERTIES: -:ID: 964c6bac-486a-4459-825f-6e7e9351c9ab -:END: -** Factor :factor: -:PROPERTIES: -:ID: 52994125-5689-402a-ac61-680aa690bf24 -:END: -- [2023-07-04 Tue] - Factor is a cool concatenative lang but unfortunately the C interface - (vm/master.h) no longer exists on the master branch. -** Joy :joy: -:PROPERTIES: -:ID: 60ca3000-2ff5-4372-93fb-d5c311fb6409 -:END: - -*** https://hypercubed.github.io/joy/html/j02maf.html -:PROPERTIES: -:ID: 56e64d52-4950-4fec-b933-73d1cd8048d1 -:END: - -*** [[https://builds.openlogicproject.org/content/incompleteness/arithmetization-syntax/arithmetization-syntax.pdf][arithmetization of syntax]] -:PROPERTIES: -:ID: 59979091-395d-4067-9c3d-e557fc5287ee -:END: -* Lisp :lisp: -:PROPERTIES: -:ID: 4b842734-135e-4c86-9337-3841c1241d3b -:END: -These notes pertain to Lisp. More specifically, ANSI Common Lisp in -most places. - -- https://github.com/lispnik/iup/ - doesn't support MacOS yet, looks - cool though - - what we really need is wasm compiler.. TBD -* Rust -:PROPERTIES: -:ID: f92ce2ba-1b66-42f4-b5e2-f1586b14760b -:END: -** Serde -:PROPERTIES: -:ID: 31512714-9c30-4144-9673-327808a18767 -:END: -- [2023-07-05 Wed] \\ - important part of the Rust ecosystem, another dtolnay - contribution. If you want to program a /data/ format in the Rust - ecosystem, this is how you do it. - - The way it works is that you define some special structs, a - Serializer and a Deserializer which implement the Serialize and - Deserialize traits provided by serde, respectively. - - You can use these structs to provide your public API. The - conventional choice is public top-level functions like from-str - and to-string. That's it, your serialization library can now read and - write your data format as Rust data types. - - [[https://serde.rs/enum-representations.html][enum-representations]] - - the default behavior is an externally tagged representation (verbose) - - The docs use strings as core IO when implementing a custom format, - but the convention is to implement for T where T is bound by std::io - Read or Write trait. Then you can provide a more robust public API - (from_bytes, from_writer, etc). -* C -:PROPERTIES: -:ID: 060042a0-d581-411a-9091-d16d11860d89 -:END: -* CPP -:PROPERTIES: -:ID: f09d7294-a997-4715-bceb-ffe6af7f3978 -:END: -* Nu -:PROPERTIES: -:ID: 765ad403-54e1-49cf-b4e6-918de9f910c2 -:END: -[[https://www.nushell.sh/][~]] -[[https://www.nushell.sh/cookbook/][cookbook]] -[[https://github.com/nushell/nu_scripts][nu_scripts]] diff -r a04ca5a66178 -r 45ac54093c09 20231101.org --- a/20231101.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,21 +0,0 @@ -* AWS usage -:PROPERTIES: -:ID: b2f8d58f-a486-4fe9-895b-504987c0f63d -:END: -We're leveraging AWS for some of our public web servers for now. It's -really not realistic to expect that my home desktop and spotty Comcast -internet can serve any production workflow. What it /is/ capable of is -a private VPN, which can communicate with AWS and other cloud VPN -depots via WireGuard ([[https://dev.to/gabrieltetzner/setting-up-a-vpn-with-wireguard-server-on-aws-ec2-4a49][article]]). - -I currently use Google Domains for nas-t.net, otom8.dev, and -rwest.io - but that business is now owned by squarespace, so I would -rather move it to Route53. - -We have archlinux ec2 image builds [[https://wiki.archlinux.org/title/Arch_Linux_AMIs_for_Amazon_Web_Services][here]] and [[https://gitlab.com/anemos-io/archlinux-ec2][here]] - only half work and not -maintained, but it's a start. I'm not even sure if I should stick with -arch or cave and use Ubuntu or AWS Linux. We can serve the static -services with little cost, the only big spender will be the heptapod -instance which requires a larger instance and some workers. - -We'll try to keep the cost at or around $30/month. diff -r a04ca5a66178 -r 45ac54093c09 20231102.org --- a/20231102.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,139 +0,0 @@ -* IDEAS -:PROPERTIES: -:ID: 836b5ca7-6c6b-4d57-8b44-3bb4a8f38a38 -:END: -** shed -:PROPERTIES: -:ID: fc9a94e1-91c5-4915-90b8-73218fa3b8bc -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:24] -:END: -rlib -> ulib -> ulib -> ulib -> ulib - -*** sh* tools -:PROPERTIES: -:ID: c0613a13-7ccb-4af9-b47e-e14a41c782c2 -:END: -:LOGBOOK: -- State "TODO" from "TODO" [2023-04-07 Fri 23:22] -:END: -shc,shx,etc -** packy -:PROPERTIES: -:ID: 6bd29c37-4b23-4a05-b3a4-72b6ee712f27 -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:33] -:END: -*** rust -:PROPERTIES: -:ID: 3bfa733e-d777-4e82-b108-93ba74c4dc08 -:END: -*** common-lisp -:PROPERTIES: -:ID: 0f3b6182-1dd1-4e65-8b1c-4630a46fff4c -:END: -*** emacs-lisp -:PROPERTIES: -:ID: e9fc19d5-daea-40a5-a0f2-882b925e58f2 -:END: -*** python -:PROPERTIES: -:ID: c2e62af0-0397-4560-a283-b33d12fbd966 -:END: -*** julia -:PROPERTIES: -:ID: d2466880-1721-4386-b385-ded06d5b15cf -:END: -*** C -:PROPERTIES: -:ID: fb722462-9e11-40c1-a5d7-30698dba7e20 -:END: -*** C++ -:PROPERTIES: -:ID: 663cd894-0a5f-4352-acf1-4528b9efeeaf -:END: -** tenex -:PROPERTIES: -:ID: b4264fa2-6e96-4679-bd17-2428ba3489ee -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:52] -:END: -** mpk -:PROPERTIES: -:ID: d9661e4a-0328-4864-878d-096f76ddaf55 -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:52] -:END: -** cfg -:PROPERTIES: -:ID: 501593d4-702a-40cb-8a8f-19edfe2f9f29 -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:34] -:END: -** obj -:PROPERTIES: -:ID: 24700a26-b238-48dd-a24f-5539984b4fa6 -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:51] -:END: -split out from rlib to separate package -- a purely OOP class library -** lab -:PROPERTIES: -:ID: fa1c9563-338b-4a56-bfc3-6f4983f2a732 -:END: -:LOGBOOK: -- State "TODO" from [2023-04-07 Fri 23:34] -:END: -** source categories -:PROPERTIES: -:ID: c43fdb5d-4f52-478e-9384-ec0e6e89cf09 -:END: -- need a way of extracting metadata from a repo -- need ability to search and query libs/packages -- separate modules based on where they belong in our stack? - - app - - lib - - script? - - dist - - software distros -** generic query language -:PROPERTIES: -:ID: 8647801a-d5ad-446d-8ae1-fd896bce129a -:END: -from obj protocol? -sql compatibility? - -/check out kdb/ -** bbdb -:PROPERTIES: -:ID: ff7c2876-e471-480b-b532-9921dbd3bc89 -:END: -:LOGBOOK: -- Note taken on [2023-10-24 Tue 22:16] \\ - graph database, build on rocksdb -:END: -insidious Big Brother database. -- an application built with obj -- sql - -** NAS-TV :nas:t: -:PROPERTIES: -:ID: 00e5fe6c-80b4-4172-a9cb-ce8adc40d3dc -:END: -:LOGBOOK: -- State "NOTE" from "TODO" [2024-08-18 Sun 18:46] -:END: -- media streaming -- gstreamer backend -- audio/video diff -r a04ca5a66178 -r 45ac54093c09 20231105.org --- a/20231105.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,187 +0,0 @@ -* DRAFT dylib-skel-1 -:PROPERTIES: -:ID: b4d1bc91-f344-45fd-becc-cb20f00a3a61 -:END: -- State "DRAFT" from [2023-11-05 Sun 22:23] -** Overview -:PROPERTIES: -:ID: 2e490c4b-344e-4790-9184-1c05ba675f15 -:END: -Our core languages are [[https://www.rust-lang.org/][Rust]] and [[https://lisp-lang.org/][Lisp]] - this is the killer combo which will allow NAS-T -to rapidly develop high-quality software. As such, it's crucial that these two very -different languages (i.e. compilers) are able to interoperate seamlessly. - -Some interop methods are easy to accomodate via the OS - such as IPC or data sharing, -but others are a bit more difficult. - -In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something -that /can/ be difficult, due to some complications with Rust and because this is not the -most popular software stack (yet ;). This is an experiment and may not make it to our -code-base, but it's definitely something worth adding to the toolbox in case we need it. - -** FFI -:PROPERTIES: -:ID: 985019fc-612a-44ab-b726-b9067432ad87 -:END: -The level of interop we're after in this case is [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFI]]. - -Basically, calling Rust code from Lisp and vice-versa. There's an article about calling -Rust from Common Lisp [[https://dev.to/veer66/calling-rust-from-common-lisp-45c5][here]] which shows the basics and serves as a great starting point -for those interested. -*** Rust != C -:PROPERTIES: -:ID: 2f71a3c1-0b14-46a6-9d8d-f6ec697729cc -:END: -The complication(s) with Rust I mentioned early is really just that /it is not C/. =C= -is old, i.e. well-supported with a stable ABI, making the process of creating bindings -for a C library a breeze in many languages. - -For a Rust library we need to first appease the compiler, as explained in [[https://doc.rust-lang.org/nomicon/ffi.html#calling-rust-code-from-c][this section]] -of the Rustonomicon. Among other things it involves changing the calling-convention of -functions with a type signature and editing the Cargo.toml file to produce a -C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used -like the C ABI can. - -*** Overhead -:PROPERTIES: -:ID: 4ea79f68-55ec-4da3-a184-8343d49532b6 -:END: -Using FFI involves some overhead. Check [[https://github.com/dyu/ffi-overhead][here]] for an example benchmark across a few -languages. While building the NAS-T core, I'm very much aware of this, and will need a -few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular, -I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp). - -** Rust -> C -> Lisp -:PROPERTIES: -:ID: a498276c-8525-4a43-aa40-4b05f76a29a9 -:END: -*** Setup -:PROPERTIES: -:ID: 19f96ef7-af92-496e-9d42-70c4d4c85051 -:END: -For starters, I'm going to assume we all have Rust (via =rustup=) and Lisp (=sbcl= only) -installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in -this post). -**** Cargo -:PROPERTIES: -:ID: c929e0b6-b6f2-4383-9412-1610329ab28c -:END: -Create a new library crate. For this example we're focusing on a 'skeleton' for -/dynamic/ libraries only, so our experiment will be called =dylib-skel= or *dysk* for -short. -src_sh[:exports code]{cargo init dysk --lib && cd dysk} - -A =src/lib.rs= will be generated for you. Go ahead and delete that. We're going to be -making our own =lib.rs= file directly in the root directory (just to be cool). - -The next step is to edit your =Cargo.toml= file. Add these lines after the =[package]= -section and before =[dependencies]=: -#+begin_src conf-toml -[lib] -crate-type = ["cdylib","rlib"] -path = "lib.rs" -[[bin]] -name="dysk-test" -path="test.rs" -#+end_src - -This tells Rust to generate a shared C-compatible object with a =.so= extension which we -can open using [[https://man.archlinux.org/man/dlopen.3.en][dlopen]]. -**** cbindgen -:PROPERTIES: -:ID: 256ac288-c5a0-473a-ab65-2d6503bd423c -:END: -***** install -:PROPERTIES: -:ID: fc476f64-6b68-417a-8540-ca23ce27fa25 -:END: -Next, we want the =cbindgen= program which we'll use to generate header files for -C/C++. This step isn't necessary at all, we just want it for further experimentation. - -src_sh[:exports code]{cargo install --force cbindgen} - -We append the =cbindgen= crate as a /build dependency/ to our =Cargo.toml= like so: -#+begin_src conf-toml -[build-dependencies] -cbindgen = "0.24" -#+end_src -***** cbindgen.toml -:PROPERTIES: -:ID: 111e27f7-0b9c-4eef-9117-f7c8ba3f511c -:END: -#+begin_src conf-toml :tangle cbindgen.toml -language = "C" -autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */" -include_version = true -namespace = "dysk" -cpp_compat = true -after_includes = "#define DYSK_VERSION \"0.1.0\"" -line_length = 88 -tab_width = 2 -documentation = true -documentation_style = "c99" -usize_is_size_t = true -[cython] -header = '"dysk.h"' -#+end_src -***** build.rs -:PROPERTIES: -:ID: 9fc271b2-9acb-4f4b-aa61-82d60d2ddb9e -:END: -#+begin_src rust :tangle build.rs -fn main() -> Result<(), cbindgen::Error> { - if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) { - b.write_to_file("dysk.h"); Ok(())} - else { panic!("failed to generate dysk.h from cbindgen.toml") } } -#+end_src -*** lib.rs -:PROPERTIES: -:ID: 6b524921-2ae0-43f0-bb85-d9955b0e689c -:END: -#+begin_src rust :tangle lib.rs -//! lib.rs --- dysk library -use std::ffi::{c_char, c_int, CString}; -#[no_mangle] -pub extern "C" fn dysk_hello() -> *const c_char { - CString::new("hello from rust").unwrap().into_raw()} -#[no_mangle] -pub extern "C" fn dysk_plus(a:c_int,b:c_int) -> c_int {a+b} -#[no_mangle] -pub extern "C" fn dysk_plus1(n:c_int) -> c_int {n+1} -#+end_src -*** test.rs -:PROPERTIES: -:ID: cc7c6538-33a6-40c6-94ef-2a9c259c975a -:END: -#+begin_src rust :tangle test.rs -//! test.rs --- dysk test -fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::dysk_plus1(2 as core::ffi::c_int);}} -#+end_src -*** compile -:PROPERTIES: -:ID: 337a24d1-f305-4e1a-9052-47a53591cb2f -:END: -#+begin_src sh -cargo build --release -#+end_src -*** load from SBCL -:PROPERTIES: -:ID: a4813269-92fb-4f52-aef0-3a36dce3cf69 -:END: -#+begin_src lisp :tangle dysk.lisp -(load-shared-object #P"target/release/libdysk.so") -(define-alien-routine dysk-hello c-string) -(define-alien-routine dysk-plus int (a int) (b int)) -(define-alien-routine dysk-plus1 int (n int)) -(dysk-hello) ;; => "hello from rust" -#+end_src -*** benchmark -:PROPERTIES: -:ID: 1a8ca441-f290-46c7-b979-1e7e0d1d063b -:END: -#+begin_src shell -time target/release/dysk-test -#+end_src -#+begin_src lisp :tangle test.lisp -(time (dotimes (_ 500000000) (dysk-plus1 2))) -#+end_src diff -r a04ca5a66178 -r 45ac54093c09 20231124.org --- a/20231124.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,36 +0,0 @@ -* cl-dot examples -:PROPERTIES: -:ID: c5f4f25c-071b-4a2d-85bc-08676eacce5d -:END: -#+begin_src lisp -(defmethod cl-dot:graph-object-node ((graph (eql 'example)) (object cons)) - (make-instance 'cl-dot:node - :attributes '(:label "cell \\N" - :shape :box))) -(defmethod cl-dot:graph-object-points-to ((graph (eql 'example)) (object cons)) - (list (car object) - (make-instance 'cl-dot:attributed - :object (cdr object) - :attributes '(:weight 3)))) -;; Symbols -(defmethod cl-dot:graph-object-node ((graph (eql 'example)) (object symbol)) - (make-instance 'cl-dot:node - :attributes `(:label ,object - :shape :hexagon - :style :filled - :color :black - :fillcolor "#ccccff"))) -(let* ((data '(a b c #1=(b z) c d #1#)) - (dgraph (cl-dot:generate-graph-from-roots 'example (list data) - '(:rankdir "LR" :layout "twopi" :labelloc "t")))) - (cl-dot:dot-graph dgraph "test-lr.svg" :format #+nil :x11 :svg)) -#+end_src - -#+RESULTS: - -#+begin_src lisp -(let* ((data '(a b)) - (dgraph (cl-dot:generate-graph-from-roots 'example (list data) - '(:rankdir "LR")))) - (cl-dot:print-graph dgraph)) -#+end_src diff -r a04ca5a66178 -r 45ac54093c09 20231205.org --- a/20231205.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,8 +0,0 @@ -* global refs -:PROPERTIES: -:ID: 7fb40057-05aa-445b-bc75-0b1b6ca2c994 -:END: -need a way of indexing, referring to, and annotating objects such as -URLs, docs, articles, source files, etc. - -What is the best way to get this done? diff -r a04ca5a66178 -r 45ac54093c09 20231209.org --- a/20231209.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,7 +0,0 @@ -* doc best practices -:PROPERTIES: -:ID: ba4a3074-f29d-4f81-b4ba-d6a523b3f291 -:END: -https://rust-lang.github.io/api-guidelines/documentation.html - -also: https://lisp-lang.org/style-guide/ diff -r a04ca5a66178 -r 45ac54093c09 20231212.org --- a/20231212.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,101 +0,0 @@ -* On Computers -:PROPERTIES: -:ID: 7845b0fb-b5d3-4488-bdba-4876d9c8816e -:END: -If you've met me in the past decade, you probably know that I am -extremely passionate about computers. Let me first explain why. - -On the most basic level computers are little (or big) machines that -can be programmed to do things, or /compute/ if we're being -technical.[fn:1] - -They host and provide access to the Internet, which is a pretty big -thing, but they do little things too like unlock your car door and -tell your microwave to beep at you. They solve problems. Big or small. - -They're also /everywhere/ - which can be scary to think about, but -ultimately helps propel us into the future. - -There's something pretty cool about that - when you look at the -essence of computation. There are endless quantities of these machines -which follow the same basic rules and can be used to solve /real/ -problems. - -** The Programmer -:PROPERTIES: -:ID: b17c853e-1728-4bd3-94c5-82e5b33d7398 -:END: -Now, let us consider the /programmer/. They have power. /real/ -power. They understand the language of computers, can whisper to them -in various dialects. It can be intimidating to witness until you -realize how often the programmer says the wrong thing - a bug. - -In reality, the programmer has a symbiotic relationship with -computers. Good programmers understand this relationship well. - -#+begin_annecdote -One day after I got my first job at a software company, I remember -being on an all-hands meeting due to a client service outage. We had -some management, our lead devs, product team, and one curious looking -man who happened to be our lead IT consultant who had just joined. He -was sitting up on a hotel bed, shirtless, vaping an e-cig, typing -away in what I can only imagine was a shell prompt. - -After several minutes he took a swig from a bottle of Coke and said -"Node 6 is sick." then a few seconds later our services were -restored. For the next hour on the call he explained what happened and -why, but that particular phrase always stuck with me. He didn't say -Node 6 was down, or had an expired cert - his diagnosis was that /it/ -was /sick/. -#+end_annecdote - -The more you work closely with computers, the more you start to think -of them this way. You don't start screaming when the computer does the -wrong thing, you figure out what's wrong and learn from it. With -experience, you start to understand the different behaviors of the -machines you work with. I like to call this /Machine Empathy/. - -** Programs -:PROPERTIES: -:ID: a297e5f9-a875-4512-b126-9a2b3e75c1d8 -:END: -I already mentioned bugs - I write plenty of those, but usually I try -to write /programs/. Programs to me are like poetry. I like to think -they are for the computer too. - -Just like computers, /computer programs/ come in different shapes and -sizes but in basic terms they are sets of instructions used to control -a computer. - -You can write programs to do anything - when I first started, my -programs made music. The program was a means to an end. Over time, I -started to see the program as something much more. I saw it as the -music itself. - -[fn:1] ... perform computations - - -* On Infra -:PROPERTIES: -:ID: 80d800de-d305-4b95-a28c-a6f19d6f7a11 -:END: -Something that is missing from many organizations big or large, is an -effective way to store and access information, even about their own -org. - -It can be difficult problem to solve - usually there's the official -one, say Microsoft Sharepoint and then the list of unofficial sources -which becomes tribal corporate hacker knowledge. Maybe the unofficial -ones are more current, or are annotated nicely, but their very -existence breaks the system. There's no longer a single source of -truth. - -My priority in this department is writing services which process and -store information from a variety of sources in a distributed knowledge -graph. The graph can later be queried to access information on-demand. - -My idea of infrastructure is in fact to build my own Cloud. Needless -to say I don't have an O365 subscription, and wherever possible I'll -be relying on hardware I have physical access to. I'm not opposed to -cloud services at large but based on principle I like to think we -shouldn't be built on them. diff -r a04ca5a66178 -r 45ac54093c09 20231223.org --- a/20231223.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,6 +0,0 @@ -* https://cal-coop.gitlab.io/utena/utena-specification/main.pdf -:PROPERTIES: -:ID: 5c3aac3a-67e1-4867-9ac4-b58a3247f9e0 -:END: -from the author of cl-decentralise2. draft specification of a -/Maximalist/ Computing System. diff -r a04ca5a66178 -r 45ac54093c09 20231224.org --- a/20231224.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,7 +0,0 @@ -* public datasets -:PROPERTIES: -:ID: 28738250-c34e-45e2-8793-f8245472d3a3 -:END: -https://github.com/awesomedata/awesome-public-datasets -https://docs.openml.org/Datasets/ -https://wiki.pathmind.com/open-datasets diff -r a04ca5a66178 -r 45ac54093c09 20231228.org --- a/20231228.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,43 +0,0 @@ -* useful internals -:PROPERTIES: -:ID: 28581a8e-ae05-45de-9744-24f447668418 -:END: -#+begin_src lisp - sb-sys:*runtime-dlhandle* - sb-fasl:+fasl-file-version+ - sb-fasl:+backend-fasl-file-implementation+ - sb-debug:print-backtrace - sb-debug:map-backtrace - sb-pretty:pprint-dispatch-table - sb-lockless: - sb-ext:simd-pack - sb-walker:define-walker-template - sb-walker:macroexpand-all - sb-walker:walk-form - sb-kernel:empty-type - sb-kernel:*eval-calls* - sb-kernel:*gc-pin-code-pages* - sb-kernel:*restart-clusters* - sb-kernel:*save-lisp-clobbered-globals* - sb-kernel:*top-level-form-p* - sb-kernel:*universal-fun-type* - sb-kernel:*universal-type* - sb-kernel:*wild-type* - sb-kernel:+simd-pack-element-types+ - (sb-vm:memory-usage) - (sb-vm:boxed-context-register) - (sb-vm:c-find-heap->arena) - (sb-vm:copy-number-to-heap) - (sb-vm:dump-arena-objects) - (sb-vm:fixnumize) - (sb-vm:rewind-arena) - (sb-vm:show-heap->arena) - (sb-vm:with/without-arena) - (sb-cltl2:{augment-environment,compiler-let,define-declaration,parse-macro}) - (sb-cltl2:{declaration-information, variable-information, function-information}) - sb-di: - sb-assem: - sb-md5: - sb-regalloc: - sb-disassem: -#+end_src diff -r a04ca5a66178 -r 45ac54093c09 20240103.org --- a/20240103.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,39 +0,0 @@ -* [[https://github.com/sigmf/SigMF][SigMF]] -:PROPERTIES: -:ID: f21a6477-6a3f-48f7-8a9e-43f5cf65286c -:END: -#+begin_quote -Sharing sets of recorded signal data is an important part of science -and engineering. It enables multiple parties to collaborate, is often -a necessary part of reproducing scientific results (a requirement of -scientific rigor), and enables sharing data with those who do not have -direct access to the equipment required to capture it. - -Unfortunately, these datasets have historically not been very -portable, and there is not an agreed upon method of sharing metadata -descriptions of the recorded data itself. This is the problem that -SigMF solves. - -By providing a standard way to describe data recordings, SigMF -facilitates the sharing of data, prevents the "bitrot" of datasets -wherein details of the capture are lost over time, and makes it -possible for different tools to operate on the same dataset, thus -enabling data portability between tools and workflows. -#+end_quote - -the-spec: https://github.com/sigmf/SigMF/blob/sigmf-v1.x/sigmf-spec.md -* [[https://www.libvolk.org/][LibVOLK]] -:PROPERTIES: -:ID: ae34135c-488e-4aff-b795-a4d0d558f22b -:END: -Vector-Optimized Library of Kernels (simd) -* [[https://docs.kernel.org/fb/framebuffer.html][/dev/fb*]] -:PROPERTIES: -:ID: 7819a970-1f37-40a6-985f-905b25a08a4a -:END: -framebuffers, used by fbgrab/fbcat program -* [[https://docs.kernel.org/block/ublk.html][ublk]] -:PROPERTIES: -:ID: 5ccaed39-7055-4a51-ab62-769a73425270 -:END: -https://github.com/ming1/ubdsrv diff -r a04ca5a66178 -r 45ac54093c09 20240104.org --- a/20240104.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,6 +0,0 @@ -goals: -make problems smaller. - -sections: -why lisp? -- doesn't need mentioning more and more diff -r a04ca5a66178 -r 45ac54093c09 20240120.org --- a/20240120.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,60 +0,0 @@ -* TODO taobench demo -:PROPERTIES: -:ID: 81ac81c3-f183-4345-aaf3-ef7ef0434947 -:END: -:LOGBOOK: -- State "TODO" from [2024-01-21 Sun 00:32] -:END: -https://github.com/audreyccheng/taobench - shouldn't have missed this :) -obviously we need to implement this using core -- in demo/bench/tao? -* TODO clap completion for nushell -:PROPERTIES: -:ID: a4b35281-83f7-4244-93e9-20e57ba31eb3 -:END: -:LOGBOOK: -- State "TODO" from [2024-01-20 Sat 23:23] -:END: -https://github.com/clap-rs/clap/tree/master/clap_complete_nushell -* Dataframe scripting -:PROPERTIES: -:ID: 8dc7339e-71c6-4a60-b285-c0d56d1cbc24 -:END: -https://studioterabyte.nl/en/blog/polars-vs-pandas -nushell supports DFs, polars underneath? -https://www.nushell.sh/book/cheat_sheet.html - -#+begin_src nushell - -#+end_src -* Cloud Squatting -:PROPERTIES: -:ID: d5131c54-9082-4424-91dd-70f17561bbf7 -:END: -** Google -:PROPERTIES: -:ID: 2b0f67de-1217-48a6-b5b7-a85d3ca15926 -:END: -- [[https://cloud.google.com/free/docs/free-cloud-features][Free Cloud Features]] - + 90-day $300 credits - + e2-micro - free hours worth 1 instance/month - + -** Amazon -:PROPERTIES: -:ID: 0fd0dedd-59b0-4b1a-9241-f65699e20c46 -:END: -- AWS Free Tier -** Akamai -:PROPERTIES: -:ID: 8315a33f-dd22-4374-979a-9cccd5b5b6a9 -:END: -- Linode Free Trial -** Oracle -:PROPERTIES: -:ID: 3f1e1116-c123-4fdf-a336-0d9b50c3dcc7 -:END: -- [[https://www.oracle.com/cloud/free/?intcmp=ohp052322ocift][OCI Free Tier]] - + always free: 2 x oracle autonomous DB - + 2 x AMD Compute VMs - + up to 4 x ARM Ampere A1 with 3k/cpu/hr and 18k/gb/h per month - + block/object/archive storage - + 30-day $300 credits diff -r a04ca5a66178 -r 45ac54093c09 20240129.org --- a/20240129.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,73 +0,0 @@ -* trash as block device -:PROPERTIES: -:ID: 4452b84f-f3a5-44f1-b60a-4efa6413cbb9 -:END: -:LOGBOOK: -- State "NOTE" from [2024-01-29 Mon 20:53] -- State "NOTE" from [2024-01-29 Mon 20:53] -:END: -in nushell there is option for rm command to always use 'trash' - -AFAIK the current approach is via a service (trashd). - -An interesting experiment would be to designate a block device as -'trash' - may be possible to remove reliance on a service - -may be an opportunity for ublk driver to shine - instead of /dev/null -piping we need a driver for streaming a file to /dev/trash -* compute power -:PROPERTIES: -:ID: aa10a1d7-f1e7-4b92-b91a-56c234d52204 -:END: -:LOGBOOK: -- State "NOTE" from [2024-01-29 Mon 16:28] -:END: -- mostly x86_64 machines - currently 2 AWS EC2 instances, some podman containers, and our home beowulf server: -- beowulf: - - Zor - - mid-size tower enclosed (Linux/Windows) - - CPU - - Intel Core i7-6700K - - 4 @ 4.0 - - GPU - - NVIDIA GeForce GTX 1060 - - 6GB - - Storage - - Samsung SSD 850: 232.9GB - - Samsung SSD 850: 465.76GB - - ST2000DM001-1ER1: 1.82TB - - WDC WD80EAZZ-00B: 7.28TB - - PSSD T7 Shield: 3.64TB - - My Passport 0820: 1.36TB - - RAM - - 16GB (2*8) [64GB max] - - DDR4 - - Jekyll - - MacBook Pro 2019 (MacOS/Darwin) - - CPU - - Intel - - 8 @ - - RAM - - 32G DDR4 - - Hyde - - Thinkpad - - CPU - - Intel - - 4 @ - - RAM - - 24G DDR3 - - Boris - - Pinephone Pro - - CPU - - 64-bit 6-core 4x ARM Cortex A53 + 2x ARM Cortex A72 - - GPU - - Mali T860MP4 - - RAM - - 4GB LPDDR4 - - pi - - Raspberry Pi 4 Model B - - CPU - - Cortex-A72 (ARM v8) 64-bit SoC - - 4 @ 1.8GHz - - RAM - - 8 GB - - DDR4 4200 diff -r a04ca5a66178 -r 45ac54093c09 20240210.org --- a/20240210.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,16 +0,0 @@ -* BigBenches -:PROPERTIES: -:ID: 87d0f5d4-c6fe-4e21-911e-2639a24e89f9 -:END: -#+name: 1trc-nu -#+begin_src nushell - let ms = '1trc/measurements-0.parquet' - dfr open $ms - | dfr group-by station - | dfr agg [ - (dfr col measure | dfr min | dfr as "min") - (dfr col measure | dfr max | dfr as "max") - (dfr col measure | dfr sum | dfr as "sum") - (dfr col measure | dfr count | dfr as "count") - ] -#+end_src diff -r a04ca5a66178 -r 45ac54093c09 20240218.org --- a/20240218.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,78 +0,0 @@ -* WL vs X -:PROPERTIES: -:ID: b4b29989-28f2-48db-ad84-cd8d08eeefdf -:END: -:LOGBOOK: -- State "NOTE" from [2024-02-18 Sun 11:55] -:END: -In the past few months there has been drama regarding Wayland vs X. It -seems to be on everyone's minds after Artem's freakout issue and the -follow up YT vids/comments. - -I admit that it made me reconsider the fitness of WL as a whole - -there was a github gist that made some scathing arguments against it. - -It's an odd debate though. I think there are many misunderstandings. - -So first off, if we look at the homepage -https://wayland.freedesktop.org/, Wayland claims it is a replacement -for X11. It now has /manifest destiny/, which in my opinion is a great -shame. - -X-pros seem to agree that Wayland has /manifest destiny/ - like if you -are building softwares that look remotely like a window system, it's a -successor to X. That's the model of doing things and there's no way -around it. - -The disagreement starts with how this destiny - of an X2 - should be -fulfilled. X-pros want a fork of X, but it's too late for -that. WL-pros want X to run on top of Wayland compositor: -https://wayland.freedesktop.org/xserver.html. - -Xwayland is a problem for me. From the project description: 'if we're -migrating away from X, it makes sense to have a good backwards -compatibility story.' Full disclosure: I have never done significant -work on Xwayland, so perhaps my opinion is unwarranted. But I have no -intention of attempting to maintain a computer system that uses -Wayland and X clients at the same time. - -To me, X is ol' reliable. Every distro has first-class X support, and -it runs on most systems with very little user intervention. Where it -doesn't, there is 20+ years of dev history and battle-tested -workarounds for you to find your solution in. - -Wayland is the new kid on the block, born just in 2008. It's a fresh -start to one of the most difficult challenges in software - window -systems. A re-write would be pointless though, and so the real -value-add is in design. Wayland is designed as a protocol and -collection of libraries which are implemented in your own -compositor. Coming from Lisp - with ANSI Common Lisp and SRFIs, this -feels right even if the implementation is something very different -(compositor vs compiler). - -With X, it is assumed to be much harder to write an equivalent -'compositor'. Here's the thing though - with a significantly complex X -client implementation, it is /impossible/ to replicate in WL. This is -really the crux of Artemi's argument in his issue. He asked for a 1:1 -equivalent X/WL comparison when no such thing exists, and in my -opinion it is a waste of time. - -The WL core team is fully aware of this dichotomy, but also that this -is in no way a problem or weakness in either system. It means they're -different systems, goddammit. - -If it was up to me, Xwayland wouldn't exist. I understand why it does, -and that it does make things easier for developers who need to support -both, and users who have multiple apps with multiple windowing -requirements. It's a bandaid though, and one that is particularly -dangerous because it re-enforces the idea that Wayland is just X2 and -that they're fully compatible. - -What interests me in the Wayland world right now is the idea of a -small, modular, full-stack Wayland compositor API. There are several -'kiosk' based compositors for single applications (cage), but these -aren't complete solutions. It is possible to get much closer to the -metal, and that's where I want to be so that I can build my own APIs -on top - I don't want to live on top of X, and I certainly don't want -to live on top of X on top of WL. I want a /pure/ solution that hides -as little as possible, exposing the interesting bits. diff -r a04ca5a66178 -r 45ac54093c09 20240301.org --- a/20240301.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,9 +0,0 @@ -* TODO collect more data -:PROPERTIES: -:ID: 69ce5703-d3fe-4df7-abf4-dc93a08ebc12 -:END: -:LOGBOOK: -- State "TODO" from [2024-03-01 Fri 15:27] -:END: -https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ -weather - music - etc diff -r a04ca5a66178 -r 45ac54093c09 20240302.org --- a/20240302.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,126 +0,0 @@ -* On blocks and devices -:PROPERTIES: -:ID: c8304ae3-415d-4ccf-8f4c-1f3f6a555e5b -:END: -:LOGBOOK: -- State "NOTE" from [2024-03-02 Sat 21:30] -:END: -[[https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/dev.html][/dev]] -In Linux, everything is a file. - -/dev/ contains special device files - usually block or character -device. - -major, minor = category, device -0, 5 - -mknod - create special device files - -[[https://www.redhat.com/en/blog/linux-block-devices-hints-debugging-and-new-developments][redhat hints]] - -#+begin_src shell - dd if=/dev/zero of=myfile bs=1M count=32 - losetup --show -f myfile - ls -al /dev/loop0 - losetup -d /dev/loop0 #teardown -#+end_src - -#+begin_src shell - echo "sup dude" > /dev/loop0 - dd if=/dev/loop0 -bs=1 - dd if=/dev/nvme0 of=/dev/null progress=true - #pacman -S hdparm - hdparm -T /dev/nvme0 -#+end_src - -#+begin_src shell -modprobe scsi_debug add_host=5 max_luns=10 num_tgts=2 dev_size_mb=16 -#+end_src - -sparsefiles: create with C, dd, or truncate - -#+begin_src shell :results output -truncate --help -#+end_src - -#+RESULTS: -#+begin_example -Usage: truncate OPTION... FILE... -Shrink or extend the size of each FILE to the specified size - -A FILE argument that does not exist is created. - -If a FILE is larger than the specified size, the extra data is lost. -If a FILE is shorter, it is extended and the sparse extended part (hole) -reads as zero bytes. - -Mandatory arguments to long options are mandatory for short options too. - -c, --no-create do not create any files - -o, --io-blocks treat SIZE as number of IO blocks instead of bytes - -r, --reference=RFILE base size on RFILE - -s, --size=SIZE set or adjust the file size by SIZE bytes - --help display this help and exit - --version output version information and exit - -The SIZE argument is an integer and optional unit (example: 10K is 10*1024). -Units are K,M,G,T,P,E,Z,Y,R,Q (powers of 1024) or KB,MB,... (powers of 1000). -Binary prefixes can be used, too: KiB=K, MiB=M, and so on. - -SIZE may also be prefixed by one of the following modifying characters: -'+' extend by, '-' reduce by, '<' at most, '>' at least, -'/' round down to multiple of, '%' round up to multiple of. - -GNU coreutils online help: -Full documentation -or available locally via: info '(coreutils) truncate invocation' -#+end_example - -test mkfs.btrfs on 10T dummy block device -#+begin_src shell - dd if=/dev/zero of=/tmp/bb1 bs=1 count=1 seek=10T - du -sh /tmp/bb1 - losetup --show -f /tmp/bb1 - mkfs.btrfs /dev/loop0 -#+end_src - -diagnostics -#+begin_src shell - iostat # pacman -S sysstat - blktrace # paru -S blktrace - iotop # pacman -S iotop -#+end_src - -bcc/ trace: Who/which process is executing specific functions against -block devices? - -bcc/biosnoop: Which process is accessing the block device, how many -bytes are accessed, which latency for answering the requests? - -at the kernel level besides BPF we got kmods and DKMS, - -compression/de-duplication can be done via VDO kernel mod - -https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support - -* save-lisp-and-respawn -:PROPERTIES: -:ID: e58a96e7-7b88-4e4a-b958-3000603d8baa -:END: -:LOGBOOK: -- State "NOTE" from [2024-03-02 Sat 22:57] -:END: -#+begin_src lisp - sb-ext:*save-hooks* -#+end_src - -* syslog for log -:PROPERTIES: -:ID: 2efa5e4a-f063-49bb-9e61-6917eb115c1a -:END: -:LOGBOOK: -- State "NOTE" from [2024-03-03 Sun 16:35] -:END: -sb-posix: -- openlog syslog closelog -- levels: emerg alert crit err warning notice info debug -- setlogmask diff -r a04ca5a66178 -r 45ac54093c09 20240313.org --- a/20240313.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,15 +0,0 @@ -* RESEARCH [[https://github.com/guicho271828/sbcl-wiki/wiki/][sbcl-wiki]] -:PROPERTIES: -:ID: 33bc81c1-c5fa-4518-9456-58fd67f301ff -:END: -:LOGBOOK: -- State "RESEARCH" from [2024-03-13 Wed 21:49] -:END: -** IR1 -:PROPERTIES: -:ID: 2c012572-ea16-408c-b741-d1f6cd0c7961 -:END: -** IR2 -:PROPERTIES: -:ID: 8d015e5e-7ad3-44c3-88cd-81e293a887e0 -:END: diff -r a04ca5a66178 -r 45ac54093c09 20240317.org --- a/20240317.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,130 +0,0 @@ -* DB Benchmarking -:PROPERTIES: -:ID: d30aec42-a9a1-425b-87ca-1e40db0e69de -:END: -:LOGBOOK: -- State "NOTE" from [2024-02-04 Sun 20:40] -:END: -[[https://github.com/facebook/rocksdb/wiki/Benchmarking-tools][RocksDB benchmarking tools]] - -* packy design -:PROPERTIES: -:ID: 76ae24f5-46e8-4b91-8991-41245383d337 -:END: -:LOGBOOK: -- State "NOTE" from [2024-01-25 Thu 22:39] -:END: -- API root: https://packy.compiler.company -- source packs: https://vc.compiler.company/packy -** Lib -:PROPERTIES: -:ID: 2d6aa6fb-a34c-4ccd-99d3-4f98d33bb2f9 -:END: -*** Types -:PROPERTIES: -:ID: 8017b6c7-3a04-4e52-b3d6-afc68cbce139 -:END: -**** Pack -:PROPERTIES: -:ID: 2722b7dc-9546-4d94-82b0-abfc02647b4d -:END: -Primary data type of the library - typically represents a compressed -archive, metadata, and ops. -**** Bundle -:PROPERTIES: -:ID: 9ec5d682-da8e-4d02-b2fc-a56a8f209c22 -:END: -Collection data type, usually contains a set of packs with metadata. -**** PackyEndpoint -:PROPERTIES: -:ID: 8cf2658a-92d5-41ab-8630-4f4e6ddb140d -:END: -Represents a Packy instance bound to a UDP socket -**** PackyEndpointConfig -:PROPERTIES: -:ID: 421b8aaa-66e1-4883-937d-fc6705146998 -:END: -Global endpoint configuration object -**** PackyClientConfig -:PROPERTIES: -:ID: c1899710-034e-4ed6-959a-1eb2df72e5f9 -:END: -Configuration for outgoing packy connections on an endpoint -**** PackyServerConfig -:PROPERTIES: -:ID: 955e8ce9-ff7e-4f42-9d25-3aeb9846528b -:END: -Configuration for incoming packy connection son an endpoint -**** PackyConnection -:PROPERTIES: -:ID: 7075e0c8-65b3-4fbf-9a6b-f27017276a72 -:END: -Packy connection object -*** Traits -:PROPERTIES: -:ID: ea92ad07-d12c-4ee8-a3d4-0955f0af3921 -:END: -**** PackyClient -:PROPERTIES: -:ID: 6c0fdd7e-bc0a-4217-b315-72fe7766e4df -:END: -***** query -:PROPERTIES: -:ID: 1358e1c5-b215-48eb-9e1b-4116cc32b2ae -:END: -***** install -:PROPERTIES: -:ID: b8298207-4d92-4628-8687-9f4e20c8cfec -:END: -***** update -:PROPERTIES: -:ID: a6fb4fb6-b188-4033-8291-107cf1f00e6d -:END: -***** login -:PROPERTIES: -:ID: 1fd91c8b-ab90-4db0-b74e-1c2b9ae2e440 -:END: -***** logout -:PROPERTIES: -:ID: 75c0db2e-37cf-4f33-95bd-3f5c95b45c49 -:END: -***** pull -:PROPERTIES: -:ID: 60de92f4-b4a6-4cae-95bc-174e2036e57b -:END: -***** push -:PROPERTIES: -:ID: e379b71e-8a84-4547-a694-90cad3b7952e -:END: -**** PackyServer -:PROPERTIES: -:ID: 73ea9478-5251-40c3-8331-cf042d0b9bf1 -:END: -***** start_packy_server -:PROPERTIES: -:ID: c8a9bac3-971a-45a7-988f-92db19ef6601 -:END: -***** stop_packy_server -:PROPERTIES: -:ID: e3122bf1-f770-46a7-a36b-2df9c48bc233 -:END: -***** start_packy_registry -:PROPERTIES: -:ID: b6d214f2-d6de-4d46-b2c2-c51ff0f3b3e6 -:END: -**** PackyRegistry -:PROPERTIES: -:ID: 0e846cbb-5cb4-496f-8100-beae6e66972e -:END: -***** register_pack -:PROPERTIES: -:ID: 764642ad-a33b-415b-9c32-8a729a7cd62d -:END: -***** register_user -:PROPERTIES: -:ID: 2b92f584-d769-4266-940e-cfbd466d4e62 -:END: -***** register_bundle -:PROPERTIES: -:ID: 390cb42b-5170-4175-9ccc-b990842356a0 -:END: diff -r a04ca5a66178 -r 45ac54093c09 20240325.org --- a/20240325.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,7 +0,0 @@ -* TBD investigate alieneval for phash opps -:PROPERTIES: -:ID: 21ab0a6e-cb66-4c5e-9b10-4ac2340ce75d -:END: -:LOGBOOK: -- State "TBD" from [2024-03-25 Mon 18:56] -:END: diff -r a04ca5a66178 -r 45ac54093c09 20240419.org --- a/20240419.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,142 +0,0 @@ -* How it works -:PROPERTIES: -:ID: 439a4945-de2f-4985-ba6c-fe6421aa9ad9 -:END: -The backend services are written in Rust and controlled by a simple -messaging protocol. Services provide common runtime capabilities known -as the /service protocol/ but are specialized on a unique /service -type/ which may in turn register their own /custom protocols/ (via -core). - -Services are capable of dispatching data directly to clients, or -storing data in the /database/ (sqlite, postgres, mysql). - -The frontend clients are pre-dominantly written in Common Lisp and -come in many shapes and sizes. There is a cli-client, web-client -(CLOG), docker-client (archlinux, stumpwm, McCLIM), and native-client -which also compiles to WASM (slint-rs). - -* Guide -:PROPERTIES: -:ID: 65140d2c-5d59-48cd-946c-01c380a37f08 -:END: -** Build -:PROPERTIES: -:ID: ad0652ca-4abf-4d27-8897-a61a13a6e61d -:END: -- *install dependencies* - #+begin_src bash - ./tools/deps.sh - #+end_src -- *make executables* \\ - Simply run =make build=. Read the ~makefile~ and change the options - as needed. -- MODE :: Mode (debug, release) -- LISP :: Lisp (sbcl, cmucl, ccl) -- CFG :: Config (default.cfg) -** Run -:PROPERTIES: -:ID: b2295442-ccb7-4a5c-8a9e-ea8502aad636 -:END: -#+begin_src shell - ./demo -i -#+end_src -** Config -:PROPERTIES: -:ID: 600be170-d032-4280-a7b5-c46e3c157206 -:END: -Configs can be specified in JSON, TOML, RON, or of course SEXP. See -=default.cfg= for an example. -** Play -:PROPERTIES: -:ID: 56a69ed7-2eb3-4490-95a1-072cee985df5 -:END: -The high-level user interface is presented as a multi-modal GUI -application which adapts to the specific application /instances/ -below. -*** Weather -:PROPERTIES: -:ID: 804a174b-534c-4a04-966a-b44c0a86fbd4 -:END: -This backend retrieves weather data using the NWS API. -*** Stocks -:PROPERTIES: -:ID: 4e7339bb-aaf3-4fab-90e4-05e13f24f881 -:END: -The 'Stocks' backend features a stock ticker with real-time analysis -capabilities. -*** Bench -:PROPERTIES: -:ID: e3cf0add-6ded-4c05-a05d-7ef463b85b08 -:END: -This is a benchmark backend for testing the capabilities of our -demo. It spins up some mock services and allows fine-grained control -of input/throughput. -* tasks -:PROPERTIES: -:ID: a16a3807-0e06-4c8d-baf6-5dfa8f8f5585 -:END: -** TODO DSLs -:PROPERTIES: -:ID: 3ae4754e-0c0f-4923-9e53-ae962b8638a1 -:END: -- consider tree-sitter parsing layout, use as a guide for developing a - single syntax which expands to Rust or C. -- with-rs -- with-c -- with-rs/c -- with-cargo -- compile-rs/c -*** TODO rs-macroexpand -:PROPERTIES: -:ID: f612975f-6bcc-4aab-b0d8-65b8ff0a0d0f -:END: -- rs-gen-file -- rs-defmacro -- rs-macros -- rs-macroexpand -- rs-macroexpand-1 -*** TODO c-macroexpand -:PROPERTIES: -:ID: 49f5c224-026d-4cdc-a326-1742e64c4279 -:END: -- c-gen-file h/c -- c-defmacro -- c-macros -- c-macroexpand -- c-macroexpand-1 -*** TODO slint-macroexpand -:PROPERTIES: -:ID: 486beb12-179b-4acc-8507-cdc498edb628 -:END: -- slint-gen-file -- slint-defmacro -- slint-macros -- slint-macroexpand -- slint-macroexpand-1 -*** TODO html (using who) -:PROPERTIES: -:ID: 2e81a19c-874c-4fd7-be0f-f6cd62c660db -:END: -** TODO web templates -:PROPERTIES: -:ID: 95f7d6ac-c010-4d1d-a47d-1693bb7d6eab -:END: -create a basic static page in CL which will be used to host Slint UIs -and other WASM doo-dads in a browser. -** TODO CLI -:PROPERTIES: -:ID: fc1d5364-279c-4d5f-baea-ed2c110b63f0 -:END: -using clingon, decide on generic options and write it up -** TODO docs -:PROPERTIES: -:ID: 467e84e3-e72f-4764-8858-cc15fdc17e8a -:END: -work on doc generation -- Rust and CL should be accounted for. -** TODO tests -:PROPERTIES: -:ID: 70aea01c-d01e-4ddf-aaf5-c86183a7e250 -:END: -We have none! need to make it more comfy - set up testing in all Rust -crates and for the lisp systems. diff -r a04ca5a66178 -r 45ac54093c09 20240425.org --- a/20240425.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,4 +0,0 @@ -* https://docs.gitlab.com/ee/administration/backup_restore/migrate_to_new_server.html -:PROPERTIES: -:ID: 264b81f4-f953-4698-9c82-d8e20b6c31f9 -:END: diff -r a04ca5a66178 -r 45ac54093c09 20240731.org --- a/20240731.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,11 +0,0 @@ -* alpine builders -:PROPERTIES: -:ID: 635925f6-e1d8-47bc-a32f-28bca0f19d2d -:END: -- make sure to apk add: - - git, hg - - clang - - make - - linux-headers - - zstd-dev - - libc-dev? diff -r a04ca5a66178 -r 45ac54093c09 20240804.org --- a/20240804.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,8 +0,0 @@ -* bookmarks -:PROPERTIES: -:ID: 6e9c6bce-0513-4628-b6f9-5c234cccdaa3 -:END: -- How should such objects be represented within CORE? -- skel/homer mostly - - already have alias -- not sure about obj/otherwise, prob not diff -r a04ca5a66178 -r 45ac54093c09 20240808.org --- a/20240808.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,12 +0,0 @@ -* Intelligent Design in Software -:PROPERTIES: -:ID: 1397d8bb-4848-4d9a-9dcc-ee65bed81c9e -:END: -- starting from a space where there are no external influences - a biome -- answer questions regarding the nature of the software and its capabilities -- incrementally adjust inter-dependencies -- optimize -- protect the biome at all costs -- focus on composition -- build applications -- re-integrate lessons learned diff -r a04ca5a66178 -r 45ac54093c09 20240816.org --- a/20240816.org Tue Aug 27 21:35:44 2024 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,11 +0,0 @@ -* keys.compiler.company -:PROPERTIES: -:ID: 86c21790-72b5-4ea1-ba49-f7be3f65a8cc -:END: -:LOGBOOK: -- State "NOTE" from [2024-08-16 Fri 22:00] -:END: -- public openpgp server -- [[https://keys.compiler.company][keys.compiler.company]] -- https://keys.openpgp.org/ -- [[vc:packy/hagrid/][packy/hagrid]] diff -r a04ca5a66178 -r 45ac54093c09 log.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/log.org Wed Sep 04 16:58:53 2024 -0400 @@ -0,0 +1,1506 @@ +#+title: log +#+author: Richard Westhaver +#+email: richard.westhaver@gmail.com +#+setupfile: ../clean.theme +* [2023-07-30 Sun] +** VC infrastructure +:PROPERTIES: +:ID: 2d40eec1-62f4-4f3d-9be7-110400808e06 +:END: +In heptapod we have a root group named =comp=, containg a variety of +subgroups. Some of these groups should be public, while others are +internal to comp members exclusively. Within each subgroup, we should +have the root group members automatically granted privileged access to +projects. This is relevant for the =startup= subgroup in particular, +where each project is potentially maintained by multiple non-root +contributors. + +We also need to consider how we will manage subrepos across the +organization. It is about time we start integrating HG bundles and +potentially mirrors. For our core VC pipeline we should have no +reliance on Git, but this may be difficult. It depends on the behavior +of HG bundles. + +Bookmarks/tags should be used for milestones in the root group and are +infrequent. They are more frequent in projects with a regular release +life-cycle. +** Approaching Webapps +:PROPERTIES: +:ID: 0a52be58-990a-47da-ba5a-a7da7192361c +:END: +I started poking around in the webapp space again so that I can launch +a landing page for NAS-T quickly. The Rust situation has improved +somewhat on the frontend side, and the axum backend stack is nice. + +This might seem like a lot of Rust and not a lot of Lisp, which it is, +but there's still room for Lisp wherever we need it. It mostly plays a +role in the backend, servicing the database and responding to requests +from the Rust edges. All of the important tests for the web APIs are +also written in Lisp. We will almost certainly use Lisp for all static +processing and HTML generation at compile-time. + +This I believe, is the appropriate way to integrate Lisp into a +cutting-edge web-app. You get the good parts of Lisp where you need +them (interactive debugging, dynamic language, REPL) and avoid the bad +parts (OOB optimization, RPS performance) in areas where the customer +would be impacted. In this domain, Lisp takes the form of a glue +rather than the bricks and mortar it sometimes appears to us as. + +* [2023-10-24 Tue] +** virt +:PROPERTIES: +:ID: a51b54d9-5663-411e-b0ba-d1405622eb88 +:END: +*** QEMU +:PROPERTIES: +:ID: 34dc9d12-d651-4204-9abb-a6a9a6aeb07e +:END: +*** KVM +:PROPERTIES: +:ID: 909e1127-14f5-4783-97af-65aa25439226 +:END: +*** Hyper-V +:PROPERTIES: +:ID: d89dc59d-25d9-4304-bcb8-379dd43cc3f7 +:END: +*** Firecracker +:PROPERTIES: +:ID: 7c4f08b7-2bc0-4a85-a6ca-2decdce8dc91 +:END: +*** Docker +:PROPERTIES: +:ID: 42ff77c2-69c9-4f89-a297-88efd592502e +:END: +*** Vagrant +:PROPERTIES: +:ID: c53c4d87-b0fd-40de-8db7-b92c22dc946c +:END: +*** LXC +:PROPERTIES: +:ID: 85c5f471-cf87-49b7-a6ce-5c1526df4080 +:END: +*** LXD +:PROPERTIES: +:ID: f92d34dd-7b24-498c-a5c8-29fd3b528092 +:END: +*** containerd +:PROPERTIES: +:ID: a967645f-5df3-4bed-9e7b-a1adaed3796c +:END: +*** systemd-nspawn +:PROPERTIES: +:ID: 6329c6f9-c54c-4eb0-99e7-4772399161e7 +:END: +*** VirtualBox +:PROPERTIES: +:ID: 638aa81a-8926-480a-854f-0434e8119426 +:END: + +** Concatenative +:PROPERTIES: +:ID: 964c6bac-486a-4459-825f-6e7e9351c9ab +:END: +*** Factor :factor: +:PROPERTIES: +:ID: 52994125-5689-402a-ac61-680aa690bf24 +:END: +- [2023-07-04 Tue] + Factor is a cool concatenative lang but unfortunately the C interface + (vm/master.h) no longer exists on the master branch. +*** Joy :joy: +:PROPERTIES: +:ID: 60ca3000-2ff5-4372-93fb-d5c311fb6409 +:END: + +**** https://hypercubed.github.io/joy/html/j02maf.html +:PROPERTIES: +:ID: 56e64d52-4950-4fec-b933-73d1cd8048d1 +:END: + +**** [[https://builds.openlogicproject.org/content/incompleteness/arithmetization-syntax/arithmetization-syntax.pdf][arithmetization of syntax]] +:PROPERTIES: +:ID: 59979091-395d-4067-9c3d-e557fc5287ee +:END: +** Lisp :lisp: +:PROPERTIES: +:ID: 4b842734-135e-4c86-9337-3841c1241d3b +:END: +These notes pertain to Lisp. More specifically, ANSI Common Lisp in +most places. + +- https://github.com/lispnik/iup/ - doesn't support MacOS yet, looks + cool though + - what we really need is wasm compiler.. TBD +** Rust +:PROPERTIES: +:ID: f92ce2ba-1b66-42f4-b5e2-f1586b14760b +:END: +*** Serde +:PROPERTIES: +:ID: 31512714-9c30-4144-9673-327808a18767 +:END: +- [2023-07-05 Wed] \\ + important part of the Rust ecosystem, another dtolnay + contribution. If you want to program a /data/ format in the Rust + ecosystem, this is how you do it. + + The way it works is that you define some special structs, a + Serializer and a Deserializer which implement the Serialize and + Deserialize traits provided by serde, respectively. + + You can use these structs to provide your public API. The + conventional choice is public top-level functions like from-str + and to-string. That's it, your serialization library can now read and + write your data format as Rust data types. + + [[https://serde.rs/enum-representations.html][enum-representations]] + - the default behavior is an externally tagged representation (verbose) + + The docs use strings as core IO when implementing a custom format, + but the convention is to implement for T where T is bound by std::io + Read or Write trait. Then you can provide a more robust public API + (from_bytes, from_writer, etc). +** C +:PROPERTIES: +:ID: 060042a0-d581-411a-9091-d16d11860d89 +:END: +** CPP +:PROPERTIES: +:ID: f09d7294-a997-4715-bceb-ffe6af7f3978 +:END: +** Nu +:PROPERTIES: +:ID: 765ad403-54e1-49cf-b4e6-918de9f910c2 +:END: +[[https://www.nushell.sh/][~]] +[[https://www.nushell.sh/cookbook/][cookbook]] +[[https://github.com/nushell/nu_scripts][nu_scripts]] +* [2023-11-01 Wed] +** AWS usage +:PROPERTIES: +:ID: b2f8d58f-a486-4fe9-895b-504987c0f63d +:END: +We're leveraging AWS for some of our public web servers for now. It's +really not realistic to expect that my home desktop and spotty Comcast +internet can serve any production workflow. What it /is/ capable of is +a private VPN, which can communicate with AWS and other cloud VPN +depots via WireGuard ([[https://dev.to/gabrieltetzner/setting-up-a-vpn-with-wireguard-server-on-aws-ec2-4a49][article]]). + +I currently use Google Domains for nas-t.net, otom8.dev, and +rwest.io - but that business is now owned by squarespace, so I would +rather move it to Route53. + +We have archlinux ec2 image builds [[https://wiki.archlinux.org/title/Arch_Linux_AMIs_for_Amazon_Web_Services][here]] and [[https://gitlab.com/anemos-io/archlinux-ec2][here]] - only half work and not +maintained, but it's a start. I'm not even sure if I should stick with +arch or cave and use Ubuntu or AWS Linux. We can serve the static +services with little cost, the only big spender will be the heptapod +instance which requires a larger instance and some workers. + +We'll try to keep the cost at or around $30/month. +* [2023-11-02 Thu] +** IDEAS +:PROPERTIES: +:ID: 836b5ca7-6c6b-4d57-8b44-3bb4a8f38a38 +:END: +*** shed +:PROPERTIES: +:ID: fc9a94e1-91c5-4915-90b8-73218fa3b8bc +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:24] +:END: +rlib +> ulib +> ulib +> ulib +> ulib + +**** sh* tools +:PROPERTIES: +:ID: c0613a13-7ccb-4af9-b47e-e14a41c782c2 +:END: +:LOGBOOK: +- State "TODO" from "TODO" [2023-04-07 Fri 23:22] +:END: +shc,shx,etc +*** packy +:PROPERTIES: +:ID: 6bd29c37-4b23-4a05-b3a4-72b6ee712f27 +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:33] +:END: +**** rust +:PROPERTIES: +:ID: 3bfa733e-d777-4e82-b108-93ba74c4dc08 +:END: +**** common-lisp +:PROPERTIES: +:ID: 0f3b6182-1dd1-4e65-8b1c-4630a46fff4c +:END: +**** emacs-lisp +:PROPERTIES: +:ID: e9fc19d5-daea-40a5-a0f2-882b925e58f2 +:END: +**** python +:PROPERTIES: +:ID: c2e62af0-0397-4560-a283-b33d12fbd966 +:END: +**** julia +:PROPERTIES: +:ID: d2466880-1721-4386-b385-ded06d5b15cf +:END: +**** C +:PROPERTIES: +:ID: fb722462-9e11-40c1-a5d7-30698dba7e20 +:END: +**** C++ +:PROPERTIES: +:ID: 663cd894-0a5f-4352-acf1-4528b9efeeaf +:END: +*** tenex +:PROPERTIES: +:ID: b4264fa2-6e96-4679-bd17-2428ba3489ee +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:52] +:END: +*** mpk +:PROPERTIES: +:ID: d9661e4a-0328-4864-878d-096f76ddaf55 +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:52] +:END: +*** cfg +:PROPERTIES: +:ID: 501593d4-702a-40cb-8a8f-19edfe2f9f29 +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:34] +:END: +*** obj +:PROPERTIES: +:ID: 24700a26-b238-48dd-a24f-5539984b4fa6 +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:51] +:END: +split out from rlib to separate package +- a purely OOP class library +*** lab +:PROPERTIES: +:ID: fa1c9563-338b-4a56-bfc3-6f4983f2a732 +:END: +:LOGBOOK: +- State "TODO" from [2023-04-07 Fri 23:34] +:END: +*** source categories +:PROPERTIES: +:ID: c43fdb5d-4f52-478e-9384-ec0e6e89cf09 +:END: +- need a way of extracting metadata from a repo +- need ability to search and query libs/packages +- separate modules based on where they belong in our stack? + - app + - lib + - script? + - dist + - software distros +*** generic query language +:PROPERTIES: +:ID: 8647801a-d5ad-446d-8ae1-fd896bce129a +:END: +from obj protocol? +sql compatibility? + +/check out kdb/ +*** bbdb +:PROPERTIES: +:ID: ff7c2876-e471-480b-b532-9921dbd3bc89 +:END: +:LOGBOOK: +- Note taken on [2023-10-24 Tue 22:16] \\ + graph database, build on rocksdb +:END: +insidious Big Brother database. +- an application built with obj +- sql + +*** NAS-TV :nas:t: +:PROPERTIES: +:ID: 00e5fe6c-80b4-4172-a9cb-ce8adc40d3dc +:END: +:LOGBOOK: +- State "NOTE" from "TODO" [2024-08-18 Sun 18:46] +:END: +- media streaming +- gstreamer backend +- audio/video +* [2023-11-05 Sun] +** DRAFT dylib-skel-1 +:PROPERTIES: +:ID: b4d1bc91-f344-45fd-becc-cb20f00a3a61 +:END: +- State "DRAFT" from [2023-11-05 Sun 22:23] +*** Overview +:PROPERTIES: +:ID: 2e490c4b-344e-4790-9184-1c05ba675f15 +:END: +Our core languages are [[https://www.rust-lang.org/][Rust]] and [[https://lisp-lang.org/][Lisp]] - this is the killer combo which will allow NAS-T +to rapidly develop high-quality software. As such, it's crucial that these two very +different languages (i.e. compilers) are able to interoperate seamlessly. + +Some interop methods are easy to accomodate via the OS - such as IPC or data sharing, +but others are a bit more difficult. + +In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something +that /can/ be difficult, due to some complications with Rust and because this is not the +most popular software stack (yet ;). This is an experiment and may not make it to our +code-base, but it's definitely something worth adding to the toolbox in case we need it. + +*** FFI +:PROPERTIES: +:ID: 985019fc-612a-44ab-b726-b9067432ad87 +:END: +The level of interop we're after in this case is [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFI]]. + +Basically, calling Rust code from Lisp and vice-versa. There's an article about calling +Rust from Common Lisp [[https://dev.to/veer66/calling-rust-from-common-lisp-45c5][here]] which shows the basics and serves as a great starting point +for those interested. +**** Rust != C +:PROPERTIES: +:ID: 2f71a3c1-0b14-46a6-9d8d-f6ec697729cc +:END: +The complication(s) with Rust I mentioned early is really just that /it is not C/. =C= +is old, i.e. well-supported with a stable ABI, making the process of creating bindings +for a C library a breeze in many languages. + +For a Rust library we need to first appease the compiler, as explained in [[https://doc.rust-lang.org/nomicon/ffi.html#calling-rust-code-from-c][this section]] +of the Rustonomicon. Among other things it involves changing the calling-convention of +functions with a type signature and editing the Cargo.toml file to produce a +C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used +like the C ABI can. + +**** Overhead +:PROPERTIES: +:ID: 4ea79f68-55ec-4da3-a184-8343d49532b6 +:END: +Using FFI involves some overhead. Check [[https://github.com/dyu/ffi-overhead][here]] for an example benchmark across a few +languages. While building the NAS-T core, I'm very much aware of this, and will need a +few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular, +I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp). + +*** Rust -> C -> Lisp +:PROPERTIES: +:ID: a498276c-8525-4a43-aa40-4b05f76a29a9 +:END: +**** Setup +:PROPERTIES: +:ID: 19f96ef7-af92-496e-9d42-70c4d4c85051 +:END: +For starters, I'm going to assume we all have Rust (via =rustup=) and Lisp (=sbcl= only) +installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in +this post). +***** Cargo +:PROPERTIES: +:ID: c929e0b6-b6f2-4383-9412-1610329ab28c +:END: +Create a new library crate. For this example we're focusing on a 'skeleton' for +/dynamic/ libraries only, so our experiment will be called =dylib-skel= or *dysk* for +short. +src_sh[:exports code]{cargo init dysk --lib && cd dysk} + +A =src/lib.rs= will be generated for you. Go ahead and delete that. We're going to be +making our own =lib.rs= file directly in the root directory (just to be cool). + +The next step is to edit your =Cargo.toml= file. Add these lines after the =[package]= +section and before =[dependencies]=: +#+begin_src conf-toml +[lib] +crate-type = ["cdylib","rlib"] +path = "lib.rs" +[[bin]] +name="dysk-test" +path="test.rs" +#+end_src + +This tells Rust to generate a shared C-compatible object with a =.so= extension which we +can open using [[https://man.archlinux.org/man/dlopen.3.en][dlopen]]. +***** cbindgen +:PROPERTIES: +:ID: 256ac288-c5a0-473a-ab65-2d6503bd423c +:END: +****** install +:PROPERTIES: +:ID: fc476f64-6b68-417a-8540-ca23ce27fa25 +:END: +Next, we want the =cbindgen= program which we'll use to generate header files for +C/C++. This step isn't necessary at all, we just want it for further experimentation. + +src_sh[:exports code]{cargo install --force cbindgen} + +We append the =cbindgen= crate as a /build dependency/ to our =Cargo.toml= like so: +#+begin_src conf-toml +[build-dependencies] +cbindgen = "0.24" +#+end_src +****** cbindgen.toml +:PROPERTIES: +:ID: 111e27f7-0b9c-4eef-9117-f7c8ba3f511c +:END: +#+begin_src conf-toml :tangle cbindgen.toml +language = "C" +autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */" +include_version = true +namespace = "dysk" +cpp_compat = true +after_includes = "#define DYSK_VERSION \"0.1.0\"" +line_length = 88 +tab_width = 2 +documentation = true +documentation_style = "c99" +usize_is_size_t = true +[cython] +header = '"dysk.h"' +#+end_src +****** build.rs +:PROPERTIES: +:ID: 9fc271b2-9acb-4f4b-aa61-82d60d2ddb9e +:END: +#+begin_src rust :tangle build.rs +fn main() -> Result<(), cbindgen::Error> { + if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) { + b.write_to_file("dysk.h"); Ok(())} + else { panic!("failed to generate dysk.h from cbindgen.toml") } } +#+end_src +**** lib.rs +:PROPERTIES: +:ID: 6b524921-2ae0-43f0-bb85-d9955b0e689c +:END: +#+begin_src rust :tangle lib.rs +//! lib.rs --- dysk library +use std::ffi::{c_char, c_int, CString}; +#[no_mangle] +pub extern "C" fn dysk_hello() -> *const c_char { + CString::new("hello from rust").unwrap().into_raw()} +#[no_mangle] +pub extern "C" fn dysk_plus(a:c_int,b:c_int) -> c_int {a+b} +#[no_mangle] +pub extern "C" fn dysk_plus1(n:c_int) -> c_int {n+1} +#+end_src +**** test.rs +:PROPERTIES: +:ID: cc7c6538-33a6-40c6-94ef-2a9c259c975a +:END: +#+begin_src rust :tangle test.rs +//! test.rs --- dysk test +fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::dysk_plus1(2 as core::ffi::c_int);}} +#+end_src +**** compile +:PROPERTIES: +:ID: 337a24d1-f305-4e1a-9052-47a53591cb2f +:END: +#+begin_src sh +cargo build --release +#+end_src +**** load from SBCL +:PROPERTIES: +:ID: a4813269-92fb-4f52-aef0-3a36dce3cf69 +:END: +#+begin_src lisp :tangle dysk.lisp +(load-shared-object #P"target/release/libdysk.so") +(define-alien-routine dysk-hello c-string) +(define-alien-routine dysk-plus int (a int) (b int)) +(define-alien-routine dysk-plus1 int (n int)) +(dysk-hello) ;; => "hello from rust" +#+end_src +**** benchmark +:PROPERTIES: +:ID: 1a8ca441-f290-46c7-b979-1e7e0d1d063b +:END: +#+begin_src shell +time target/release/dysk-test +#+end_src +#+begin_src lisp :tangle test.lisp +(time (dotimes (_ 500000000) (dysk-plus1 2))) +#+end_src +* [2023-11-24 Fri] +** cl-dot examples +:PROPERTIES: +:ID: c5f4f25c-071b-4a2d-85bc-08676eacce5d +:END: +#+begin_src lisp +(defmethod cl-dot:graph-object-node ((graph (eql 'example)) (object cons)) + (make-instance 'cl-dot:node + :attributes '(:label "cell \\N" + :shape :box))) +(defmethod cl-dot:graph-object-points-to ((graph (eql 'example)) (object cons)) + (list (car object) + (make-instance 'cl-dot:attributed + :object (cdr object) + :attributes '(:weight 3)))) +;; Symbols +(defmethod cl-dot:graph-object-node ((graph (eql 'example)) (object symbol)) + (make-instance 'cl-dot:node + :attributes `(:label ,object + :shape :hexagon + :style :filled + :color :black + :fillcolor "#ccccff"))) +(let* ((data '(a b c #1=(b z) c d #1#)) + (dgraph (cl-dot:generate-graph-from-roots 'example (list data) + '(:rankdir "LR" :layout "twopi" :labelloc "t")))) + (cl-dot:dot-graph dgraph "test-lr.svg" :format #+nil :x11 :svg)) +#+end_src + +#+RESULTS: + +#+begin_src lisp +(let* ((data '(a b)) + (dgraph (cl-dot:generate-graph-from-roots 'example (list data) + '(:rankdir "LR")))) + (cl-dot:print-graph dgraph)) +#+end_src + +* [2023-12-05 Tue] +** global refs +:PROPERTIES: +:ID: 7fb40057-05aa-445b-bc75-0b1b6ca2c994 +:END: +need a way of indexing, referring to, and annotating objects such as +URLs, docs, articles, source files, etc. + +What is the best way to get this done? +* [2023-12-09 Sat] +** doc best practices +:PROPERTIES: +:ID: ba4a3074-f29d-4f81-b4ba-d6a523b3f291 +:END: +https://rust-lang.github.io/api-guidelines/documentation.html + +also: https://lisp-lang.org/style-guide/ +* [2023-12-12 Tue] +** On Computers +:PROPERTIES: +:ID: 7845b0fb-b5d3-4488-bdba-4876d9c8816e +:END: +If you've met me in the past decade, you probably know that I am +extremely passionate about computers. Let me first explain why. + +On the most basic level computers are little (or big) machines that +can be programmed to do things, or /compute/ if we're being +technical.[fn:1] + +They host and provide access to the Internet, which is a pretty big +thing, but they do little things too like unlock your car door and +tell your microwave to beep at you. They solve problems. Big or small. + +They're also /everywhere/ - which can be scary to think about, but +ultimately helps propel us into the future. + +There's something pretty cool about that - when you look at the +essence of computation. There are endless quantities of these machines +which follow the same basic rules and can be used to solve /real/ +problems. + +*** The Programmer +:PROPERTIES: +:ID: b17c853e-1728-4bd3-94c5-82e5b33d7398 +:END: +Now, let us consider the /programmer/. They have power. /real/ +power. They understand the language of computers, can whisper to them +in various dialects. It can be intimidating to witness until you +realize how often the programmer says the wrong thing - a bug. + +In reality, the programmer has a symbiotic relationship with +computers. Good programmers understand this relationship well. + +#+begin_annecdote +One day after I got my first job at a software company, I remember +being on an all-hands meeting due to a client service outage. We had +some management, our lead devs, product team, and one curious looking +man who happened to be our lead IT consultant who had just joined. He +was sitting up on a hotel bed, shirtless, vaping an e-cig, typing +away in what I can only imagine was a shell prompt. + +After several minutes he took a swig from a bottle of Coke and said +"Node 6 is sick." then a few seconds later our services were +restored. For the next hour on the call he explained what happened and +why, but that particular phrase always stuck with me. He didn't say +Node 6 was down, or had an expired cert - his diagnosis was that /it/ +was /sick/. +#+end_annecdote + +The more you work closely with computers, the more you start to think +of them this way. You don't start screaming when the computer does the +wrong thing, you figure out what's wrong and learn from it. With +experience, you start to understand the different behaviors of the +machines you work with. I like to call this /Machine Empathy/. + +*** Programs +:PROPERTIES: +:ID: a297e5f9-a875-4512-b126-9a2b3e75c1d8 +:END: +I already mentioned bugs - I write plenty of those, but usually I try +to write /programs/. Programs to me are like poetry. I like to think +they are for the computer too. + +Just like computers, /computer programs/ come in different shapes and +sizes but in basic terms they are sets of instructions used to control +a computer. + +You can write programs to do anything - when I first started, my +programs made music. The program was a means to an end. Over time, I +started to see the program as something much more. I saw it as the +music itself. + +[fn:1] ... perform computations + +** On Infra +:PROPERTIES: +:ID: 80d800de-d305-4b95-a28c-a6f19d6f7a11 +:END: +Something that is missing from many organizations big or large, is an +effective way to store and access information, even about their own +org. + +It can be difficult problem to solve - usually there's the official +one, say Microsoft Sharepoint and then the list of unofficial sources +which becomes tribal corporate hacker knowledge. Maybe the unofficial +ones are more current, or are annotated nicely, but their very +existence breaks the system. There's no longer a single source of +truth. + +My priority in this department is writing services which process and +store information from a variety of sources in a distributed knowledge +graph. The graph can later be queried to access information on-demand. + +My idea of infrastructure is in fact to build my own Cloud. Needless +to say I don't have an O365 subscription, and wherever possible I'll +be relying on hardware I have physical access to. I'm not opposed to +cloud services at large but based on principle I like to think we +shouldn't be built on them. +* [2023-12-23 Sat] +** https://cal-coop.gitlab.io/utena/utena-specification/main.pdf +:PROPERTIES: +:ID: 5c3aac3a-67e1-4867-9ac4-b58a3247f9e0 +:END: +from the author of cl-decentralise2. draft specification of a +/Maximalist/ Computing System. +* [2023-12-24 Sun] +** public datasets +:PROPERTIES: +:ID: 28738250-c34e-45e2-8793-f8245472d3a3 +:END: +https://github.com/awesomedata/awesome-public-datasets +https://docs.openml.org/Datasets/ +https://wiki.pathmind.com/open-datasets +* [2023-12-28 Thu] +** useful internals +:PROPERTIES: +:ID: 28581a8e-ae05-45de-9744-24f447668418 +:END: +#+begin_src lisp + sb-sys:*runtime-dlhandle* + sb-fasl:+fasl-file-version+ + sb-fasl:+backend-fasl-file-implementation+ + sb-debug:print-backtrace + sb-debug:map-backtrace + sb-pretty:pprint-dispatch-table + sb-lockless: + sb-ext:simd-pack + sb-walker:define-walker-template + sb-walker:macroexpand-all + sb-walker:walk-form + sb-kernel:empty-type + sb-kernel:*eval-calls* + sb-kernel:*gc-pin-code-pages* + sb-kernel:*restart-clusters* + sb-kernel:*save-lisp-clobbered-globals* + sb-kernel:*top-level-form-p* + sb-kernel:*universal-fun-type* + sb-kernel:*universal-type* + sb-kernel:*wild-type* + sb-kernel:+simd-pack-element-types+ + (sb-vm:memory-usage) + (sb-vm:boxed-context-register) + (sb-vm:c-find-heap->arena) + (sb-vm:copy-number-to-heap) + (sb-vm:dump-arena-objects) + (sb-vm:fixnumize) + (sb-vm:rewind-arena) + (sb-vm:show-heap->arena) + (sb-vm:with/without-arena) + (sb-cltl2:{augment-environment,compiler-let,define-declaration,parse-macro}) + (sb-cltl2:{declaration-information, variable-information, function-information}) + sb-di: + sb-assem: + sb-md5: + sb-regalloc: + sb-disassem: +#+end_src +* [2024-01-03 Wed] +** [[https://github.com/sigmf/SigMF][SigMF]] +:PROPERTIES: +:ID: f21a6477-6a3f-48f7-8a9e-43f5cf65286c +:END: +#+begin_quote +Sharing sets of recorded signal data is an important part of science +and engineering. It enables multiple parties to collaborate, is often +a necessary part of reproducing scientific results (a requirement of +scientific rigor), and enables sharing data with those who do not have +direct access to the equipment required to capture it. + +Unfortunately, these datasets have historically not been very +portable, and there is not an agreed upon method of sharing metadata +descriptions of the recorded data itself. This is the problem that +SigMF solves. + +By providing a standard way to describe data recordings, SigMF +facilitates the sharing of data, prevents the "bitrot" of datasets +wherein details of the capture are lost over time, and makes it +possible for different tools to operate on the same dataset, thus +enabling data portability between tools and workflows. +#+end_quote + +the-spec: https://github.com/sigmf/SigMF/blob/sigmf-v1.x/sigmf-spec.md +** [[https://www.libvolk.org/][LibVOLK]] +:PROPERTIES: +:ID: ae34135c-488e-4aff-b795-a4d0d558f22b +:END: +Vector-Optimized Library of Kernels (simd) +** [[https://docs.kernel.org/fb/framebuffer.html][/dev/fb*]] +:PROPERTIES: +:ID: 7819a970-1f37-40a6-985f-905b25a08a4a +:END: +framebuffers, used by fbgrab/fbcat program +** [[https://docs.kernel.org/block/ublk.html][ublk]] +:PROPERTIES: +:ID: 5ccaed39-7055-4a51-ab62-769a73425270 +:END: +https://github.com/ming1/ubdsrv +* [2024-01-04 Thu] +goals: +make problems smaller. + +sections: +why lisp? +- doesn't need mentioning more and more +* [2024-01-20 Sat] +** TODO taobench demo +:PROPERTIES: +:ID: 81ac81c3-f183-4345-aaf3-ef7ef0434947 +:END: +:LOGBOOK: +- State "TODO" from [2024-01-21 Sun 00:32] +:END: +https://github.com/audreyccheng/taobench - shouldn't have missed this :) +obviously we need to implement this using core -- in demo/bench/tao? +** TODO clap completion for nushell +:PROPERTIES: +:ID: a4b35281-83f7-4244-93e9-20e57ba31eb3 +:END: +:LOGBOOK: +- State "TODO" from [2024-01-20 Sat 23:23] +:END: +https://github.com/clap-rs/clap/tree/master/clap_complete_nushell +** Dataframe scripting +:PROPERTIES: +:ID: 8dc7339e-71c6-4a60-b285-c0d56d1cbc24 +:END: +https://studioterabyte.nl/en/blog/polars-vs-pandas +nushell supports DFs, polars underneath? +https://www.nushell.sh/book/cheat_sheet.html + +#+begin_src nushell + +#+end_src +** Cloud Squatting +:PROPERTIES: +:ID: d5131c54-9082-4424-91dd-70f17561bbf7 +:END: +*** Google +:PROPERTIES: +:ID: 2b0f67de-1217-48a6-b5b7-a85d3ca15926 +:END: +- [[https://cloud.google.com/free/docs/free-cloud-features][Free Cloud Features]] + + 90-day $300 credits + + e2-micro - free hours worth 1 instance/month + + +*** Amazon +:PROPERTIES: +:ID: 0fd0dedd-59b0-4b1a-9241-f65699e20c46 +:END: +- AWS Free Tier +*** Akamai +:PROPERTIES: +:ID: 8315a33f-dd22-4374-979a-9cccd5b5b6a9 +:END: +- Linode Free Trial +*** Oracle +:PROPERTIES: +:ID: 3f1e1116-c123-4fdf-a336-0d9b50c3dcc7 +:END: +- [[https://www.oracle.com/cloud/free/?intcmp=ohp052322ocift][OCI Free Tier]] + + always free: 2 x oracle autonomous DB + + 2 x AMD Compute VMs + + up to 4 x ARM Ampere A1 with 3k/cpu/hr and 18k/gb/h per month + + block/object/archive storage + + 30-day $300 credits +* [2024-01-29 Mon] +** trash as block device +:PROPERTIES: +:ID: 4452b84f-f3a5-44f1-b60a-4efa6413cbb9 +:END: +:LOGBOOK: +- State "NOTE" from [2024-01-29 Mon 20:53] +- State "NOTE" from [2024-01-29 Mon 20:53] +:END: +in nushell there is option for rm command to always use 'trash' - +AFAIK the current approach is via a service (trashd). + +An interesting experiment would be to designate a block device as +'trash' - may be possible to remove reliance on a service + +may be an opportunity for ublk driver to shine - instead of /dev/null +piping we need a driver for streaming a file to /dev/trash +** compute power +:PROPERTIES: +:ID: aa10a1d7-f1e7-4b92-b91a-56c234d52204 +:END: +:LOGBOOK: +- State "NOTE" from [2024-01-29 Mon 16:28] +:END: +- mostly x86_64 machines - currently 2 AWS EC2 instances, some podman containers, and our home beowulf server: +- beowulf: + - Zor + - mid-size tower enclosed (Linux/Windows) + - CPU + - Intel Core i7-6700K + - 4 @ 4.0 + - GPU + - NVIDIA GeForce GTX 1060 + - 6GB + - Storage + - Samsung SSD 850: 232.9GB + - Samsung SSD 850: 465.76GB + - ST2000DM001-1ER1: 1.82TB + - WDC WD80EAZZ-00B: 7.28TB + - PSSD T7 Shield: 3.64TB + - My Passport 0820: 1.36TB + - RAM + - 16GB (2*8) [64GB max] + - DDR4 + - Jekyll + - MacBook Pro 2019 (MacOS/Darwin) + - CPU + - Intel + - 8 @ + - RAM + - 32G DDR4 + - Hyde + - Thinkpad + - CPU + - Intel + - 4 @ + - RAM + - 24G DDR3 + - Boris + - Pinephone Pro + - CPU + - 64-bit 6-core 4x ARM Cortex A53 + 2x ARM Cortex A72 + - GPU + - Mali T860MP4 + - RAM + - 4GB LPDDR4 + - pi + - Raspberry Pi 4 Model B + - CPU + - Cortex-A72 (ARM v8) 64-bit SoC + - 4 @ 1.8GHz + - RAM + - 8 GB + - DDR4 4200 +* [2024-02-10 Sat] +** BigBenches +:PROPERTIES: +:ID: 87d0f5d4-c6fe-4e21-911e-2639a24e89f9 +:END: +#+name: 1trc-nu +#+begin_src nushell + let ms = '1trc/measurements-0.parquet' + dfr open $ms + | dfr group-by station + | dfr agg [ + (dfr col measure | dfr min | dfr as "min") + (dfr col measure | dfr max | dfr as "max") + (dfr col measure | dfr sum | dfr as "sum") + (dfr col measure | dfr count | dfr as "count") + ] +#+end_src +* [2024-02-18 Sun] +** WL vs X +:PROPERTIES: +:ID: b4b29989-28f2-48db-ad84-cd8d08eeefdf +:END: +:LOGBOOK: +- State "NOTE" from [2024-02-18 Sun 11:55] +:END: +In the past few months there has been drama regarding Wayland vs X. It +seems to be on everyone's minds after Artem's freakout issue and the +follow up YT vids/comments. + +I admit that it made me reconsider the fitness of WL as a whole - +there was a github gist that made some scathing arguments against it. + +It's an odd debate though. I think there are many misunderstandings. + +So first off, if we look at the homepage +https://wayland.freedesktop.org/, Wayland claims it is a replacement +for X11. It now has /manifest destiny/, which in my opinion is a great +shame. + +X-pros seem to agree that Wayland has /manifest destiny/ - like if you +are building softwares that look remotely like a window system, it's a +successor to X. That's the model of doing things and there's no way +around it. + +The disagreement starts with how this destiny - of an X2 - should be +fulfilled. X-pros want a fork of X, but it's too late for +that. WL-pros want X to run on top of Wayland compositor: +https://wayland.freedesktop.org/xserver.html. + +Xwayland is a problem for me. From the project description: 'if we're +migrating away from X, it makes sense to have a good backwards +compatibility story.' Full disclosure: I have never done significant +work on Xwayland, so perhaps my opinion is unwarranted. But I have no +intention of attempting to maintain a computer system that uses +Wayland and X clients at the same time. + +To me, X is ol' reliable. Every distro has first-class X support, and +it runs on most systems with very little user intervention. Where it +doesn't, there is 20+ years of dev history and battle-tested +workarounds for you to find your solution in. + +Wayland is the new kid on the block, born just in 2008. It's a fresh +start to one of the most difficult challenges in software - window +systems. A re-write would be pointless though, and so the real +value-add is in design. Wayland is designed as a protocol and +collection of libraries which are implemented in your own +compositor. Coming from Lisp - with ANSI Common Lisp and SRFIs, this +feels right even if the implementation is something very different +(compositor vs compiler). + +With X, it is assumed to be much harder to write an equivalent +'compositor'. Here's the thing though - with a significantly complex X +client implementation, it is /impossible/ to replicate in WL. This is +really the crux of Artemi's argument in his issue. He asked for a 1:1 +equivalent X/WL comparison when no such thing exists, and in my +opinion it is a waste of time. + +The WL core team is fully aware of this dichotomy, but also that this +is in no way a problem or weakness in either system. It means they're +different systems, goddammit. + +If it was up to me, Xwayland wouldn't exist. I understand why it does, +and that it does make things easier for developers who need to support +both, and users who have multiple apps with multiple windowing +requirements. It's a bandaid though, and one that is particularly +dangerous because it re-enforces the idea that Wayland is just X2 and +that they're fully compatible. + +What interests me in the Wayland world right now is the idea of a +small, modular, full-stack Wayland compositor API. There are several +'kiosk' based compositors for single applications (cage), but these +aren't complete solutions. It is possible to get much closer to the +metal, and that's where I want to be so that I can build my own APIs +on top - I don't want to live on top of X, and I certainly don't want +to live on top of X on top of WL. I want a /pure/ solution that hides +as little as possible, exposing the interesting bits. +* [2024-03-01 Fri] +** TODO collect more data +:PROPERTIES: +:ID: 69ce5703-d3fe-4df7-abf4-dc93a08ebc12 +:END: +:LOGBOOK: +- State "TODO" from [2024-03-01 Fri 15:27] +:END: +https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ +weather - music - etc +* [2024-03-02 Sat] +** On blocks and devices +:PROPERTIES: +:ID: c8304ae3-415d-4ccf-8f4c-1f3f6a555e5b +:END: +:LOGBOOK: +- State "NOTE" from [2024-03-02 Sat 21:30] +:END: +[[https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/dev.html][/dev]] +In Linux, everything is a file. + +/dev/ contains special device files - usually block or character +device. + +major, minor = category, device +0, 5 + +mknod - create special device files + +[[https://www.redhat.com/en/blog/linux-block-devices-hints-debugging-and-new-developments][redhat hints]] + +#+begin_src shell + dd if=/dev/zero of=myfile bs=1M count=32 + losetup --show -f myfile + ls -al /dev/loop0 + losetup -d /dev/loop0 #teardown +#+end_src + +#+begin_src shell + echo "sup dude" > /dev/loop0 + dd if=/dev/loop0 -bs=1 + dd if=/dev/nvme0 of=/dev/null progress=true + #pacman -S hdparm + hdparm -T /dev/nvme0 +#+end_src + +#+begin_src shell +modprobe scsi_debug add_host=5 max_luns=10 num_tgts=2 dev_size_mb=16 +#+end_src + +sparsefiles: create with C, dd, or truncate + +#+begin_src shell :results output +truncate --help +#+end_src + +#+RESULTS: +#+begin_example +Usage: truncate OPTION... FILE... +Shrink or extend the size of each FILE to the specified size + +A FILE argument that does not exist is created. + +If a FILE is larger than the specified size, the extra data is lost. +If a FILE is shorter, it is extended and the sparse extended part (hole) +reads as zero bytes. + +Mandatory arguments to long options are mandatory for short options too. + -c, --no-create do not create any files + -o, --io-blocks treat SIZE as number of IO blocks instead of bytes + -r, --reference=RFILE base size on RFILE + -s, --size=SIZE set or adjust the file size by SIZE bytes + --help display this help and exit + --version output version information and exit + +The SIZE argument is an integer and optional unit (example: 10K is 10*1024). +Units are K,M,G,T,P,E,Z,Y,R,Q (powers of 1024) or KB,MB,... (powers of 1000). +Binary prefixes can be used, too: KiB=K, MiB=M, and so on. + +SIZE may also be prefixed by one of the following modifying characters: +'+' extend by, '-' reduce by, '<' at most, '>' at least, +'/' round down to multiple of, '%' round up to multiple of. + +GNU coreutils online help: +Full documentation +or available locally via: info '(coreutils) truncate invocation' +#+end_example + +test mkfs.btrfs on 10T dummy block device +#+begin_src shell + dd if=/dev/zero of=/tmp/bb1 bs=1 count=1 seek=10T + du -sh /tmp/bb1 + losetup --show -f /tmp/bb1 + mkfs.btrfs /dev/loop0 +#+end_src + +diagnostics +#+begin_src shell + iostat # pacman -S sysstat + blktrace # paru -S blktrace + iotop # pacman -S iotop +#+end_src + +bcc/ trace: Who/which process is executing specific functions against +block devices? + +bcc/biosnoop: Which process is accessing the block device, how many +bytes are accessed, which latency for answering the requests? + +at the kernel level besides BPF we got kmods and DKMS, + +compression/de-duplication can be done via VDO kernel mod + +https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support + +** save-lisp-and-respawn +:PROPERTIES: +:ID: e58a96e7-7b88-4e4a-b958-3000603d8baa +:END: +:LOGBOOK: +- State "NOTE" from [2024-03-02 Sat 22:57] +:END: +#+begin_src lisp + sb-ext:*save-hooks* +#+end_src + +** syslog for log +:PROPERTIES: +:ID: 2efa5e4a-f063-49bb-9e61-6917eb115c1a +:END: +:LOGBOOK: +- State "NOTE" from [2024-03-03 Sun 16:35] +:END: +sb-posix: +- openlog syslog closelog +- levels: emerg alert crit err warning notice info debug +- setlogmask +* [2024-03-13 Wed] +** RESEARCH [[https://github.com/guicho271828/sbcl-wiki/wiki/][sbcl-wiki]] +:PROPERTIES: +:ID: 33bc81c1-c5fa-4518-9456-58fd67f301ff +:END: +:LOGBOOK: +- State "RESEARCH" from [2024-03-13 Wed 21:49] +:END: +** IR1 +:PROPERTIES: +:ID: 2c012572-ea16-408c-b741-d1f6cd0c7961 +:END: +** IR2 +:PROPERTIES: +:ID: 8d015e5e-7ad3-44c3-88cd-81e293a887e0 +:END: +* [2024-03-17 Sun] +** DB Benchmarking +:PROPERTIES: +:ID: d30aec42-a9a1-425b-87ca-1e40db0e69de +:END: +:LOGBOOK: +- State "NOTE" from [2024-02-04 Sun 20:40] +:END: +[[https://github.com/facebook/rocksdb/wiki/Benchmarking-tools][RocksDB benchmarking tools]] + +** packy design +:PROPERTIES: +:ID: 76ae24f5-46e8-4b91-8991-41245383d337 +:END: +:LOGBOOK: +- State "NOTE" from [2024-01-25 Thu 22:39] +:END: +- API root: https://packy.compiler.company +- source packs: https://vc.compiler.company/packy +*** Lib +:PROPERTIES: +:ID: 2d6aa6fb-a34c-4ccd-99d3-4f98d33bb2f9 +:END: +**** Types +:PROPERTIES: +:ID: 8017b6c7-3a04-4e52-b3d6-afc68cbce139 +:END: +***** Pack +:PROPERTIES: +:ID: 2722b7dc-9546-4d94-82b0-abfc02647b4d +:END: +Primary data type of the library - typically represents a compressed +archive, metadata, and ops. +***** Bundle +:PROPERTIES: +:ID: 9ec5d682-da8e-4d02-b2fc-a56a8f209c22 +:END: +Collection data type, usually contains a set of packs with metadata. +***** PackyEndpoint +:PROPERTIES: +:ID: 8cf2658a-92d5-41ab-8630-4f4e6ddb140d +:END: +Represents a Packy instance bound to a UDP socket +***** PackyEndpointConfig +:PROPERTIES: +:ID: 421b8aaa-66e1-4883-937d-fc6705146998 +:END: +Global endpoint configuration object +***** PackyClientConfig +:PROPERTIES: +:ID: c1899710-034e-4ed6-959a-1eb2df72e5f9 +:END: +Configuration for outgoing packy connections on an endpoint +***** PackyServerConfig +:PROPERTIES: +:ID: 955e8ce9-ff7e-4f42-9d25-3aeb9846528b +:END: +Configuration for incoming packy connection son an endpoint +***** PackyConnection +:PROPERTIES: +:ID: 7075e0c8-65b3-4fbf-9a6b-f27017276a72 +:END: +Packy connection object +**** Traits +:PROPERTIES: +:ID: ea92ad07-d12c-4ee8-a3d4-0955f0af3921 +:END: +***** PackyClient +:PROPERTIES: +:ID: 6c0fdd7e-bc0a-4217-b315-72fe7766e4df +:END: +****** query +:PROPERTIES: +:ID: 1358e1c5-b215-48eb-9e1b-4116cc32b2ae +:END: +****** install +:PROPERTIES: +:ID: b8298207-4d92-4628-8687-9f4e20c8cfec +:END: +****** update +:PROPERTIES: +:ID: a6fb4fb6-b188-4033-8291-107cf1f00e6d +:END: +****** login +:PROPERTIES: +:ID: 1fd91c8b-ab90-4db0-b74e-1c2b9ae2e440 +:END: +****** logout +:PROPERTIES: +:ID: 75c0db2e-37cf-4f33-95bd-3f5c95b45c49 +:END: +****** pull +:PROPERTIES: +:ID: 60de92f4-b4a6-4cae-95bc-174e2036e57b +:END: +****** push +:PROPERTIES: +:ID: e379b71e-8a84-4547-a694-90cad3b7952e +:END: +***** PackyServer +:PROPERTIES: +:ID: 73ea9478-5251-40c3-8331-cf042d0b9bf1 +:END: +****** start_packy_server +:PROPERTIES: +:ID: c8a9bac3-971a-45a7-988f-92db19ef6601 +:END: +****** stop_packy_server +:PROPERTIES: +:ID: e3122bf1-f770-46a7-a36b-2df9c48bc233 +:END: +****** start_packy_registry +:PROPERTIES: +:ID: b6d214f2-d6de-4d46-b2c2-c51ff0f3b3e6 +:END: +***** PackyRegistry +:PROPERTIES: +:ID: 0e846cbb-5cb4-496f-8100-beae6e66972e +:END: +****** register_pack +:PROPERTIES: +:ID: 764642ad-a33b-415b-9c32-8a729a7cd62d +:END: +****** register_user +:PROPERTIES: +:ID: 2b92f584-d769-4266-940e-cfbd466d4e62 +:END: +****** register_bundle +:PROPERTIES: +:ID: 390cb42b-5170-4175-9ccc-b990842356a0 +:END: +* [2024-03-25 Mon] +** TBD investigate alieneval for phash opps +:PROPERTIES: +:ID: 21ab0a6e-cb66-4c5e-9b10-4ac2340ce75d +:END: +:LOGBOOK: +- State "TBD" from [2024-03-25 Mon 18:56] +:END: +* [2024-04-19 Fri] +** How it works +:PROPERTIES: +:ID: 439a4945-de2f-4985-ba6c-fe6421aa9ad9 +:END: +The backend services are written in Rust and controlled by a simple +messaging protocol. Services provide common runtime capabilities known +as the /service protocol/ but are specialized on a unique /service +type/ which may in turn register their own /custom protocols/ (via +core). + +Services are capable of dispatching data directly to clients, or +storing data in the /database/ (sqlite, postgres, mysql). + +The frontend clients are pre-dominantly written in Common Lisp and +come in many shapes and sizes. There is a cli-client, web-client +(CLOG), docker-client (archlinux, stumpwm, McCLIM), and native-client +which also compiles to WASM (slint-rs). + +** Guide +:PROPERTIES: +:ID: 65140d2c-5d59-48cd-946c-01c380a37f08 +:END: +*** Build +:PROPERTIES: +:ID: ad0652ca-4abf-4d27-8897-a61a13a6e61d +:END: +- *install dependencies* + #+begin_src bash + ./tools/deps.sh + #+end_src +- *make executables* \\ + Simply run =make build=. Read the ~makefile~ and change the options + as needed. +- MODE :: Mode (debug, release) +- LISP :: Lisp (sbcl, cmucl, ccl) +- CFG :: Config (default.cfg) +*** Run +:PROPERTIES: +:ID: b2295442-ccb7-4a5c-8a9e-ea8502aad636 +:END: +#+begin_src shell + ./demo -i +#+end_src +*** Config +:PROPERTIES: +:ID: 600be170-d032-4280-a7b5-c46e3c157206 +:END: +Configs can be specified in JSON, TOML, RON, or of course SEXP. See +=default.cfg= for an example. +*** Play +:PROPERTIES: +:ID: 56a69ed7-2eb3-4490-95a1-072cee985df5 +:END: +The high-level user interface is presented as a multi-modal GUI +application which adapts to the specific application /instances/ +below. +**** Weather +:PROPERTIES: +:ID: 804a174b-534c-4a04-966a-b44c0a86fbd4 +:END: +This backend retrieves weather data using the NWS API. +**** Stocks +:PROPERTIES: +:ID: 4e7339bb-aaf3-4fab-90e4-05e13f24f881 +:END: +The 'Stocks' backend features a stock ticker with real-time analysis +capabilities. +**** Bench +:PROPERTIES: +:ID: e3cf0add-6ded-4c05-a05d-7ef463b85b08 +:END: +This is a benchmark backend for testing the capabilities of our +demo. It spins up some mock services and allows fine-grained control +of input/throughput. +** tasks +:PROPERTIES: +:ID: a16a3807-0e06-4c8d-baf6-5dfa8f8f5585 +:END: +*** TODO DSLs +:PROPERTIES: +:ID: 3ae4754e-0c0f-4923-9e53-ae962b8638a1 +:END: +- consider tree-sitter parsing layout, use as a guide for developing a + single syntax which expands to Rust or C. +- with-rs +- with-c +- with-rs/c +- with-cargo +- compile-rs/c +**** TODO rs-macroexpand +:PROPERTIES: +:ID: f612975f-6bcc-4aab-b0d8-65b8ff0a0d0f +:END: +- rs-gen-file +- rs-defmacro +- rs-macros +- rs-macroexpand +- rs-macroexpand-1 +**** TODO c-macroexpand +:PROPERTIES: +:ID: 49f5c224-026d-4cdc-a326-1742e64c4279 +:END: +- c-gen-file h/c +- c-defmacro +- c-macros +- c-macroexpand +- c-macroexpand-1 +**** TODO slint-macroexpand +:PROPERTIES: +:ID: 486beb12-179b-4acc-8507-cdc498edb628 +:END: +- slint-gen-file +- slint-defmacro +- slint-macros +- slint-macroexpand +- slint-macroexpand-1 +**** TODO html (using who) +:PROPERTIES: +:ID: 2e81a19c-874c-4fd7-be0f-f6cd62c660db +:END: +*** TODO web templates +:PROPERTIES: +:ID: 95f7d6ac-c010-4d1d-a47d-1693bb7d6eab +:END: +create a basic static page in CL which will be used to host Slint UIs +and other WASM doo-dads in a browser. +*** TODO CLI +:PROPERTIES: +:ID: fc1d5364-279c-4d5f-baea-ed2c110b63f0 +:END: +using clingon, decide on generic options and write it up +*** TODO docs +:PROPERTIES: +:ID: 467e84e3-e72f-4764-8858-cc15fdc17e8a +:END: +work on doc generation -- Rust and CL should be accounted for. +*** TODO tests +:PROPERTIES: +:ID: 70aea01c-d01e-4ddf-aaf5-c86183a7e250 +:END: +We have none! need to make it more comfy - set up testing in all Rust +crates and for the lisp systems. +* [2024-04-25 Thu] +** https://docs.gitlab.com/ee/administration/backup_restore/migrate_to_new_server.html +:PROPERTIES: +:ID: 264b81f4-f953-4698-9c82-d8e20b6c31f9 +:END: +* [2024-07-31 Wed] +** alpine builders +:PROPERTIES: +:ID: 635925f6-e1d8-47bc-a32f-28bca0f19d2d +:END: +- make sure to apk add: + - git, hg + - clang + - make + - linux-headers + - zstd-dev + - libc-dev? +* [2024-08-04 Sun] +** bookmarks +:PROPERTIES: +:ID: 6e9c6bce-0513-4628-b6f9-5c234cccdaa3 +:END: +- How should such objects be represented within CORE? +- skel/homer mostly + - already have alias +- not sure about obj/otherwise, prob not +* [2024-08-08 Thu] +** Intelligent Design in Software +:PROPERTIES: +:ID: 1397d8bb-4848-4d9a-9dcc-ee65bed81c9e +:END: +- starting from a space where there are no external influences - a biome +- answer questions regarding the nature of the software and its capabilities +- incrementally adjust inter-dependencies +- optimize +- protect the biome at all costs +- focus on composition +- build applications +- re-integrate lessons learned +* [2024-08-16 Fri] +** keys.compiler.company +:PROPERTIES: +:ID: 86c21790-72b5-4ea1-ba49-f7be3f65a8cc +:END: +:LOGBOOK: +- State "NOTE" from [2024-08-16 Fri 22:00] +:END: +- public openpgp server +- [[https://keys.compiler.company][keys.compiler.company]] +- https://keys.openpgp.org/ +- [[vc:packy/hagrid/][packy/hagrid]] diff -r a04ca5a66178 -r 45ac54093c09 readme.org --- a/readme.org Tue Aug 27 21:35:44 2024 -0400 +++ b/readme.org Wed Sep 04 16:58:53 2024 -0400 @@ -1,44 +1,18 @@ +:PROPERTIES: +:ID: 751af716-6308-4433-8f2d-78cb4c59c665 +:END: #+title: notes #+description: CC Notes #+author: Richard Westhaver #+email:richard.westhaver@gmail.com #+setupfile: ../clean.theme #+export_file_name: index -#+options: num:t prop:t timestamp:t toc:t d:t compact-itemx:t +#+options: num:nil prop:nil timestamp:nil toc:nil html-postamble:nil ^:nil #+property: header-args :eval no-export -#+INCLUDE: 20230730.org -#+INCLUDE: 20231024.org -#+INCLUDE: 20231101.org -#+INCLUDE: 20231102.org -#+INCLUDE: 20231105.org -#+INCLUDE: 20231124.org -#+INCLUDE: 20231205.org -#+INCLUDE: 20231209.org -#+INCLUDE: 20231212.org -#+INCLUDE: 20231223.org -#+INCLUDE: 20231224.org -#+INCLUDE: 20231228.org -#+INCLUDE: 20240103.org -#+INCLUDE: 20240104.org -#+INCLUDE: 20240120.org -#+INCLUDE: 20240129.org -#+INCLUDE: 20240210.org -#+INCLUDE: 20240218.org -#+INCLUDE: 20240301.org -#+INCLUDE: 20240302.org -#+INCLUDE: 20240313.org -#+INCLUDE: 20240317.org -#+INCLUDE: 20240325.org -#+INCLUDE: 20240419.org -#+INCLUDE: 20240425.org -#+INCLUDE: 20240731.org -#+INCLUDE: 20240804.org -#+INCLUDE: 20240808.org -#+INCLUDE: 20240816.org -#+INCLUDE: demo-notes.org -#+INCLUDE: inbox-notes.org -#+INCLUDE: nas-t.org -#+INCLUDE: parquet-parsing.org -#+INCLUDE: query-lang.org -#+INCLUDE: refs.bib -#+INCLUDE: skel-readme.org +* [[file:log.org][log]] +* [[file:demo-notes.org][demo-notes]] +* [[file:inbox-notes.org][inbox-notes]] +* [[file:nas-t.org][nas-t]] +* [[file:parquet-parsing.org][parquet-parsing]] +* [[file:query-lang.org][query-lang]] +* [[file:skel-readme.org][skel-readme]] diff -r a04ca5a66178 -r 45ac54093c09 skel-readme.org --- a/skel-readme.org Tue Aug 27 21:35:44 2024 -0400 +++ b/skel-readme.org Wed Sep 04 16:58:53 2024 -0400 @@ -283,7 +283,7 @@ - TODO :: CLOS-based core classes - TODO :: EIEIO-based wrapper classes -#+begin_src dot :file api.svg :exports results +#+begin_src dot :file /tmp/api.svg :exports results digraph { splines=true; label="CLOS API"; labelloc="t"; node [shape=record]; sk [label="(skel :ID :AST)"] methods [label="(sk-compile sk-expand sk-build\nsk-run sk-init sk-new sk-save\nsk-tangle sk-weave sk-call sk-print)"] @@ -325,4 +325,4 @@ #+end_src #+RESULTS: -[[file:api.svg]] +[[file:/tmp/api.svg]]