#+title: log

 #+author: Richard Westhaver

 #+email: richard.westhaver@gmail.com

 #+setupfile: ../clean.theme

 * [2023-07-30 Sun]

 :PROPERTIES:

 :ID:       5a1f17a0-22bb-45bc-a313-ec2cb6c0bfd5

 :END:

 ** VC infrastructure

 :PROPERTIES:

 :ID:       9928339e-14d7-425d-a465-be87e2784590

 :END:

 In heptapod we have a root group named =comp=, containg a variety of

 subgroups. Some of these groups should be public, while others are

 internal to comp members exclusively. Within each subgroup, we should

 have the root group members automatically granted privileged access to

 projects. This is relevant for the =startup= subgroup in particular,

 where each project is potentially maintained by multiple non-root

 contributors.

 We also need to consider how we will manage subrepos across the

 organization. It is about time we start integrating HG bundles and

 potentially mirrors. For our core VC pipeline we should have no

 reliance on Git, but this may be difficult. It depends on the behavior

 of HG bundles.

 Bookmarks/tags should be used for milestones in the root group and are

 infrequent. They are more frequent in projects with a regular release

 life-cycle.

 ** Approaching Webapps

 :PROPERTIES:

 :ID:       bafaceb9-2798-484d-80b7-7ad5d84fff1a

 :END:

 I started poking around in the webapp space again so that I can launch

 a landing page for NAS-T quickly. The Rust situation has improved

 somewhat on the frontend side, and the axum backend stack is nice.

 This might seem like a lot of Rust and not a lot of Lisp, which it is,

 but there's still room for Lisp wherever we need it. It mostly plays a

 role in the backend, servicing the database and responding to requests

 from the Rust edges. All of the important tests for the web APIs are

 also written in Lisp. We will almost certainly use Lisp for all static

 processing and HTML generation at compile-time.

 This I believe, is the appropriate way to integrate Lisp into a

 cutting-edge web-app. You get the good parts of Lisp where you need

 them (interactive debugging, dynamic language, REPL) and avoid the bad

 parts (OOB optimization, RPS performance) in areas where the customer

 would be impacted. In this domain, Lisp takes the form of a glue

 rather than the bricks and mortar it sometimes appears to us as.

 * [2023-10-24 Tue]

 :PROPERTIES:

 :ID:       7bd213b5-9a4b-4490-b0b8-8a0a9b42a9c6

 :END:

 ** virt

 :PROPERTIES:

 :ID:       07c7016f-c653-4d69-9ca0-366a7584a212

 :END:

 *** QEMU

 :PROPERTIES:

 :ID:       f53afecb-561e-4545-a6d0-dbf6d9504ec4

 :END:

 *** KVM

 :PROPERTIES:

 :ID:       df9e07cd-986f-45a7-b9f7-ea88313c92ee

 :END:

 *** Hyper-V

 :PROPERTIES:

 :ID:       87b019ff-7f05-496d-b7bb-153fea7f8595

 :END:

 *** Firecracker

 :PROPERTIES:

 :ID:       b1c52088-4244-4342-b884-c551ce43cc9c

 :END:

 *** Docker

 :PROPERTIES:

 :ID:       d1969d15-8dc0-4499-bddb-f92fa19aec13

 :END:

 *** Vagrant

 :PROPERTIES:

 :ID:       dd6e8169-fba6-45af-aaef-6acd9dbd7b3f

 :END:

 *** LXC

 :PROPERTIES:

 :ID:       bd393c26-6014-42bb-a425-b9f72e5cee8b

 :END:

 *** LXD

 :PROPERTIES:

 :ID:       800ee282-7b5c-465f-a372-344079f9d875

 :END:

 *** containerd

 :PROPERTIES:

 :ID:       7f34835c-9135-4015-bdbd-99d28d7b40bb

 :END:

 *** systemd-nspawn

 :PROPERTIES:

 :ID:       9fef114a-e2f3-40b4-bddb-9948b47b441f

 :END:

 *** VirtualBox

 :PROPERTIES:

 :ID:       04594cbd-5847-434d-aaf6-a7e23058d980

 :END:

 ** Concatenative

 :PROPERTIES:

 :ID:       ad2041de-3915-48f9-950c-03b01968952d

 :END:

 *** Factor                                                           :factor:

 :PROPERTIES:

 :ID:       edb7e4a8-d07d-4953-9b64-decdca1dd76f

 :END:

 - [2023-07-04 Tue]

   Factor is a cool concatenative lang but unfortunately the C interface

   (vm/master.h) no longer exists on the master branch.

 *** Joy                                                                 :joy:

 :PROPERTIES:

 :ID:       257365d1-1b2e-4abd-a2e3-5d1705e41bcf

 :END:

 **** https://hypercubed.github.io/joy/html/j02maf.html

 :PROPERTIES:

 :ID:       a685ec2b-fbdc-4724-ab22-12d8a4498f81

 :END:

 **** [[https://builds.openlogicproject.org/content/incompleteness/arithmetization-syntax/arithmetization-syntax.pdf][arithmetization of syntax]]

 :PROPERTIES:

 :ID:       895a803a-63d5-4dfb-8ceb-c20627d97b02

 :END:

 ** Lisp                                                                :lisp:

 :PROPERTIES:

 :ID:       8109c42f-ccbb-4a08-ab99-1dced8b22a9f

 :END:

 These notes pertain to Lisp. More specifically, ANSI Common Lisp in

 most places.

 - https://github.com/lispnik/iup/ - doesn't support MacOS yet, looks

   cool though

   - what we really need is wasm compiler.. TBD

 ** Rust

 :PROPERTIES:

 :ID:       94a9a2b7-3559-476c-9edf-66b3b0c8cbb0

 :END:

 *** Serde

 :PROPERTIES:

 :ID:       ffb4e87c-e265-4549-91f9-093e714708c0

 :END:

 - [2023-07-05 Wed] \\

   important part of the Rust ecosystem, another dtolnay

   contribution. If you want to program a /data/ format in the Rust

   ecosystem, this is how you do it.

   The way it works is that you define some special structs, a

   Serializer and a Deserializer which implement the Serialize and

   Deserialize traits provided by serde, respectively.

   You can use these structs to provide your public API. The

   conventional choice is public top-level functions like from-str

   and to-string. That's it, your serialization library can now read and

   write your data format as Rust data types.

   [[https://serde.rs/enum-representations.html][enum-representations]]

   - the default behavior is an externally tagged representation (verbose)

   The docs use strings as core IO when implementing a custom format,

   but the convention is to implement for T where T is bound by std::io

   Read or Write trait. Then you can provide a more robust public API

   (from_bytes, from_writer, etc).

 ** C

 :PROPERTIES:

 :ID:       05a7192a-4c05-44d9-b859-e2a0cff53d86

 :END:

 ** CPP

 :PROPERTIES:

 :ID:       6800e89d-f49e-4ffe-bbab-ed10fec43174

 :END:

 ** Nu

 :PROPERTIES:

 :ID:       b4048114-ec39-4e53-816f-2f1319d404ee

 :END:

 [[https://www.nushell.sh/][~]]

 [[https://www.nushell.sh/cookbook/][cookbook]]

 [[https://github.com/nushell/nu_scripts][nu_scripts]]

 * [2023-11-01 Wed]

 :PROPERTIES:

 :ID:       6d461ba3-f8cb-4a69-b172-038efcca923a

 :END:

 ** AWS usage

 :PROPERTIES:

 :ID:       a14a7916-6743-45e5-8711-ecded8092131

 :END:

 We're leveraging AWS for some of our public web servers for now. It's

 really not realistic to expect that my home desktop and spotty Comcast

 internet can serve any production workflow. What it /is/ capable of is

 a private VPN, which can communicate with AWS and other cloud VPN

 depots via WireGuard ([[https://dev.to/gabrieltetzner/setting-up-a-vpn-with-wireguard-server-on-aws-ec2-4a49][article]]).

 I currently use Google Domains for nas-t.net, otom8.dev, and

 rwest.io - but that business is now owned by squarespace, so I would

 rather move it to Route53.

 We have archlinux ec2 image builds [[https://wiki.archlinux.org/title/Arch_Linux_AMIs_for_Amazon_Web_Services][here]] and [[https://gitlab.com/anemos-io/archlinux-ec2][here]] - only half work and not

 maintained, but it's a start. I'm not even sure if I should stick with

 arch or cave and use Ubuntu or AWS Linux. We can serve the static

 services with little cost, the only big spender will be the heptapod

 instance which requires a larger instance and some workers.

 We'll try to keep the cost at or around $30/month.

 * [2023-11-02 Thu]

 :PROPERTIES:

 :ID:       b74a60a5-fdc1-4ef2-ac74-7ff5c1395d0c

 :END:

 ** IDEAS

 :PROPERTIES:

 :ID:       a0d84027-06ea-4080-a40c-8fa86ed7ee7a

 :END:

 *** shed

 :PROPERTIES:

 :ID:       9fd5e6e1-bb06-4204-a5ed-01afb5d45379

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:24]

 :END:

 rlib

 > ulib

 > ulib

 > ulib

 > ulib

 **** sh* tools

 :PROPERTIES:

 :ID:       3daae7ec-4d0a-4f3e-bd5a-5202707fa614

 :END:

 :LOGBOOK:

 - State "TODO"       from "TODO"       [2023-04-07 Fri 23:22]

 :END:

 shc,shx,etc

 *** packy

 :PROPERTIES:

 :ID:       6b2b99d7-9d45-49fc-bff2-463a82c45f90

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:33]

 :END:

 **** rust

 :PROPERTIES:

 :ID:       0fbbf620-c3b7-4aa8-8a79-b25beca802d0

 :END:

 **** common-lisp

 :PROPERTIES:

 :ID:       7fd63e2e-532b-4742-9d25-6cacbfbf2ca0

 :END:

 **** emacs-lisp

 :PROPERTIES:

 :ID:       2e6a146a-9a81-4b49-9127-646838f5e137

 :END:

 **** python

 :PROPERTIES:

 :ID:       2d634517-84dc-4cf4-a95a-02ae2fe3c2cd

 :END:

 **** julia

 :PROPERTIES:

 :ID:       3d75df13-227c-4aee-a046-1443acdcf619

 :END:

 **** C

 :PROPERTIES:

 :ID:       8d44b6f9-e628-4983-aaa9-fea25117a303

 :END:

 **** C++

 :PROPERTIES:

 :ID:       7c0febaa-468c-4835-83ac-8280faa067b9

 :END:

 *** tenex

 :PROPERTIES:

 :ID:       4e5e71e4-be74-44d0-9221-74de37f11096

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:52]

 :END:

 *** mpk

 :PROPERTIES:

 :ID:       f9200923-f406-4104-9efe-33efab186f90

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:52]

 :END:

 *** cfg

 :PROPERTIES:

 :ID:       57a76ed9-76b9-442e-9a8e-3a7deb63dec4

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:34]

 :END:

 *** obj

 :PROPERTIES:

 :ID:       ce1ef5c1-61e0-4bed-84ff-bdf05d1cb379

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:51]

 :END:

 split out from rlib to separate package

 - a purely OOP class library

 *** lab

 :PROPERTIES:

 :ID:       a9401df7-b77e-4648-bdcf-bc9f8d267faf

 :END:

 :LOGBOOK:

 - State "TODO"       from              [2023-04-07 Fri 23:34]

 :END:

 *** source categories

 :PROPERTIES:

 :ID:       bdfaf877-a8a8-47c0-83b2-7ae3d18fcb8c

 :END:

 - need a way of extracting metadata from a repo

 - need ability to search and query libs/packages

 - separate modules based on where they belong in our stack?

   - app

   - lib

   - script?

   - dist

     - software distros

 *** generic query language

 :PROPERTIES:

 :ID:       4cbbb5bd-6fb0-4249-bce3-c60137a0545d

 :END:

 from obj protocol?

 sql compatibility?

 /check out kdb/

 *** bbdb

 :PROPERTIES:

 :ID:       2ef1a2d7-19f5-4d65-a4aa-2f6abf07a202

 :END:

 :LOGBOOK:

 - Note taken on [2023-10-24 Tue 22:16] \\

   graph database, build on rocksdb

 :END:

 insidious Big Brother database.

 - an application built with obj

 - sql

 *** NAS-TV                                                            :nas:t:

 :PROPERTIES:

 :ID:       416d5836-fd77-4f7f-ad5c-831f20802a14

 :END:

 :LOGBOOK:

 - State "NOTE"       from "TODO"       [2024-08-18 Sun 18:46]

 :END:

 - media streaming

 - gstreamer backend

 - audio/video

 * [2023-11-05 Sun]

 :PROPERTIES:

 :ID:       195067ad-1c7b-498a-9a90-8c9abba192f5

 :END:

 ** DRAFT dylib-skel-1

 :PROPERTIES:

 :ID:       55fc90c8-4261-4fd2-b991-280e76f0e59c

 :END:

 - State "DRAFT"      from              [2023-11-05 Sun 22:23]

 *** Overview

 :PROPERTIES:

 :ID:       bd3b748b-e792-40b3-abb0-02c274038539

 :END:

 Our core languages are [[https://www.rust-lang.org/][Rust]] and [[https://lisp-lang.org/][Lisp]] - this is the killer combo which will allow NAS-T

 to rapidly develop high-quality software. As such, it's crucial that these two very

 different languages (i.e. compilers) are able to interoperate seamlessly.

 Some interop methods are easy to accomodate via the OS - such as IPC or data sharing,

 but others are a bit more difficult.

 In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something

 that /can/ be difficult, due to some complications with Rust and because this is not the

 most popular software stack (yet ;). This is an experiment and may not make it to our

 code-base, but it's definitely something worth adding to the toolbox in case we need it.

 *** FFI

 :PROPERTIES:

 :ID:       c2ccef54-e93d-406a-af1d-ebb7ad1a935b

 :END:

 The level of interop we're after in this case is [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFI]].

 Basically, calling Rust code from Lisp and vice-versa. There's an article about calling

 Rust from Common Lisp [[https://dev.to/veer66/calling-rust-from-common-lisp-45c5][here]] which shows the basics and serves as a great starting point

 for those interested.

 **** Rust != C

 :PROPERTIES:

 :ID:       bb68152d-1794-47d8-86c0-03eb7f9256d7

 :END:

 The complication(s) with Rust I mentioned early is really just that /it is not C/. =C=

 is old, i.e. well-supported with a stable ABI, making the process of creating bindings

 for a C library a breeze in many languages.

 For a Rust library we need to first appease the compiler, as explained in [[https://doc.rust-lang.org/nomicon/ffi.html#calling-rust-code-from-c][this section]]

 of the Rustonomicon. Among other things it involves changing the calling-convention of

 functions with a type signature and editing the Cargo.toml file to produce a

 C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used

 like the C ABI can.

 **** Overhead

 :PROPERTIES:

 :ID:       f74790d6-8fa9-4e76-8362-71f074058947

 :END:

 Using FFI involves some overhead. Check [[https://github.com/dyu/ffi-overhead][here]] for an example benchmark across a few

 languages. While building the NAS-T core, I'm very much aware of this, and will need a

 few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular,

 I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp).

 *** Rust -> C -> Lisp

 :PROPERTIES:

 :ID:       3d8e8664-337b-4bf1-832d-bc29e9761d56

 :END:

 **** Setup

 :PROPERTIES:

 :ID:       7c1a259a-8bc1-4a42-abb7-8d61d0a1c22d

 :END:

 For starters, I'm going to assume we all have Rust (via =rustup=) and Lisp (=sbcl= only)

 installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in

 this post).

 ***** Cargo

 :PROPERTIES:

 :ID:       21b8d6b0-27fd-460d-b584-55765c5f7761

 :END:

 Create a new library crate. For this example we're focusing on a 'skeleton' for

 /dynamic/ libraries only, so our experiment will be called =dylib-skel= or *dysk* for

 short.

 src_sh[:exports code]{cargo init dysk --lib && cd dysk} 

 A =src/lib.rs= will be generated for you. Go ahead and delete that. We're going to be

 making our own =lib.rs= file directly in the root directory (just to be cool).

 The next step is to edit your =Cargo.toml= file. Add these lines after the =[package]=

 section and before =[dependencies]=:

 #+begin_src conf-toml

 [lib]

 crate-type = ["cdylib","rlib"]

 path = "lib.rs"

 [[bin]]

 name="dysk-test"

 path="test.rs"

 #+end_src

 This tells Rust to generate a shared C-compatible object with a =.so= extension which we

 can open using [[https://man.archlinux.org/man/dlopen.3.en][dlopen]].

 ***** cbindgen

 :PROPERTIES:

 :ID:       45c0e1cd-a441-4877-8b64-d30bc0381da2

 :END:

 ****** install

 :PROPERTIES:

 :ID:       d11805bf-1238-4359-ab52-6e6d1b74dc33

 :END:

 Next, we want the =cbindgen= program which we'll use to generate header files for

 C/C++. This step isn't necessary at all, we just want it for further experimentation.

 src_sh[:exports code]{cargo install --force cbindgen}

 We append the =cbindgen= crate as a /build dependency/ to our =Cargo.toml= like so:

 #+begin_src conf-toml

 [build-dependencies]

 cbindgen = "0.24"

 #+end_src

 ****** cbindgen.toml

 :PROPERTIES:

 :ID:       e33424c1-a328-4e86-b817-86e171584944

 :END:

 #+begin_src conf-toml :tangle cbindgen.toml

 language = "C"

 autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */"

 include_version = true

 namespace = "dysk"

 cpp_compat = true

 after_includes = "#define DYSK_VERSION \"0.1.0\""

 line_length = 88

 tab_width = 2

 documentation = true

 documentation_style = "c99"

 usize_is_size_t = true

 [cython]

 header = '"dysk.h"'

 #+end_src

 ****** build.rs

 :PROPERTIES:

 :ID:       0e465a4e-6709-4920-bb8a-514189f36f84

 :END:

 #+begin_src rust :tangle build.rs

 fn main() -> Result<(), cbindgen::Error> {

   if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) {

     b.write_to_file("dysk.h"); Ok(())}

   else { panic!("failed to generate dysk.h from cbindgen.toml") } }

 #+end_src

 **** lib.rs

 :PROPERTIES:

 :ID:       884ae229-0b41-4de3-880b-fb31058dc00a

 :END:

 #+begin_src rust :tangle lib.rs

 //! lib.rs --- dysk library

 use std::ffi::{c_char, c_int, CString};

 #[no_mangle]

 pub extern "C" fn dysk_hello() -> *const c_char {

   CString::new("hello from rust").unwrap().into_raw()}

 #[no_mangle]

 pub extern "C" fn dysk_plus(a:c_int,b:c_int) -> c_int {a+b}

 #[no_mangle]

 pub extern "C" fn dysk_plus1(n:c_int) -> c_int {n+1}

 #+end_src

 **** test.rs

 :PROPERTIES:

 :ID:       b8b27643-3c41-412c-afb7-12e980ffffb3

 :END:

 #+begin_src rust :tangle test.rs

 //! test.rs --- dysk test

 fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::dysk_plus1(2 as core::ffi::c_int);}}

 #+end_src

 **** compile

 :PROPERTIES:

 :ID:       4d429e91-1b3b-40eb-86c0-b4896e277423

 :END: