2 #+author: Richard Westhaver
3 #+email: richard.westhaver@gmail.com
4 #+setupfile: ../clean.theme
8 :ID: 2d40eec1-62f4-4f3d-9be7-110400808e06 10 In heptapod we have a root group named
=comp=, containg a variety of
11 subgroups. Some of these groups should be public, while others are
12 internal to comp members exclusively. Within each subgroup, we should
13 have the root group members automatically granted privileged access to
14 projects. This is relevant for the
=startup= subgroup in particular,
15 where each project is potentially maintained by multiple non-root
18 We also need to consider how we will manage subrepos across the
19 organization. It is about time we start integrating HG bundles and
20 potentially mirrors. For our core VC pipeline we should have no
21 reliance on Git, but this may be difficult. It depends on the behavior
24 Bookmarks/tags should be used for milestones in the root group and are
25 infrequent. They are more frequent in projects with a regular release
27 ** Approaching Webapps 29 :ID: 0a52be58-990a-47da-ba5a-a7da7192361c 31 I started poking around in the webapp space again so that I can launch
32 a landing page for NAS-T quickly. The Rust situation has improved
33 somewhat on the frontend side, and the axum backend stack is nice.
35 This might seem like a lot of Rust and not a lot of Lisp, which it is,
36 but there's still room for Lisp wherever we need it. It mostly plays a
37 role in the backend, servicing the database and responding to requests
38 from the Rust edges. All of the important tests for the web APIs are
39 also written in Lisp. We will almost certainly use Lisp for all static
40 processing and HTML generation at compile-time.
42 This I believe, is the appropriate way to integrate Lisp into a
43 cutting-edge web-app. You get the good parts of Lisp where you need
44 them (interactive debugging, dynamic language, REPL) and avoid the bad
45 parts (OOB optimization, RPS performance) in areas where the customer
46 would be impacted. In this domain, Lisp takes the form of a glue
47 rather than the bricks and mortar it sometimes appears to us as.
52 :ID: a51b54d9-5663-411e-b0ba-d1405622eb88 56 :ID: 34dc9d12-d651-4204-9abb-a6a9a6aeb07e 60 :ID: 909e1127-14f5-4783-97af-65aa25439226 64 :ID: d89dc59d-25d9-4304-bcb8-379dd43cc3f7 68 :ID: 7c4f08b7-2bc0-4a85-a6ca-2decdce8dc91 72 :ID: 42ff77c2-69c9-4f89-a297-88efd592502e 76 :ID: c53c4d87-b0fd-40de-8db7-b92c22dc946c 80 :ID: 85c5f471-cf87-49b7-a6ce-5c1526df4080 84 :ID: f92d34dd-7b24-498c-a5c8-29fd3b528092 88 :ID: a967645f-5df3-4bed-9e7b-a1adaed3796c 92 :ID: 6329c6f9-c54c-4eb0-99e7-4772399161e7 96 :ID: 638aa81a-8926-480a-854f-0434e8119426 101 :ID: 964c6bac-486a-4459-825f-6e7e9351c9ab 105 :ID: 52994125-5689-402a-ac61-680aa690bf24 108 Factor is a cool concatenative lang but unfortunately the C interface
109 (vm/master.h) no longer exists on the master branch.
112 :ID: 60ca3000-2ff5-4372-93fb-d5c311fb6409 115 **** https://hypercubed.github.io/joy/html/j02maf.html 117 :ID: 56e64d52-4950-4fec-b933-73d1cd8048d1 120 **** [[https://builds.openlogicproject.org/content/incompleteness/arithmetization-syntax/arithmetization-syntax.pdf][arithmetization of syntax]] 122 :ID: 59979091-395d-4067-9c3d-e557fc5287ee 126 :ID: 4b842734-135e-4c86-9337-3841c1241d3b 128 These notes pertain to Lisp. More specifically, ANSI Common Lisp in
131 - https:
//github.com/lispnik/iup/ - doesn't support MacOS yet, looks
133 - what we really need is wasm compiler.. TBD
136 :ID: f92ce2ba-1b66-42f4-b5e2-f1586b14760b 140 :ID: 31512714-9c30-4144-9673-327808a18767 142 - [2023-07-05 Wed]
\\ 143 important part of the Rust ecosystem, another dtolnay
144 contribution. If you want to program a
/data/ format in the Rust
145 ecosystem, this is how you do it.
147 The way it works is that you define some special structs, a
148 Serializer and a Deserializer which implement the Serialize and
149 Deserialize traits provided by serde, respectively.
151 You can use these structs to provide your public API. The
152 conventional choice is public top-level functions like from-str
153 and to-string. That's it, your serialization library can now read and
154 write your data format as Rust data types.
156 [[https://serde.rs/enum-representations.html][enum-representations]] 157 - the default behavior is an externally tagged representation (verbose)
159 The docs use strings as core IO when implementing a custom format,
160 but the convention is to implement for T where T is bound by std::io
161 Read or Write trait. Then you can provide a more robust public API
162 (from_bytes, from_writer, etc).
165 :ID: 060042a0-d581-411a-9091-d16d11860d89 169 :ID: f09d7294-a997-4715-bceb-ffe6af7f3978 173 :ID: 765ad403-54e1-49cf-b4e6-918de9f910c2 175 [[https://www.nushell.sh/][~]] 176 [[https://www.nushell.sh/cookbook/][cookbook]] 177 [[https://github.com/nushell/nu_scripts][nu_scripts]] 181 :ID: b2f8d58f-a486-4fe9-895b-504987c0f63d 183 We're leveraging AWS for some of our public web servers for now. It's
184 really not realistic to expect that my home desktop and spotty Comcast
185 internet can serve any production workflow. What it
/is/ capable of is
186 a private VPN, which can communicate with AWS and other cloud VPN
187 depots via WireGuard (
[[https://dev.to/gabrieltetzner/setting-up-a-vpn-with-wireguard-server-on-aws-ec2-4a49][article]]).
189 I currently use Google Domains for nas-t.net, otom8.dev, and
190 rwest.io - but that business is now owned by squarespace, so I would
191 rather move it to Route53.
193 We have archlinux ec2 image builds
[[https://wiki.archlinux.org/title/Arch_Linux_AMIs_for_Amazon_Web_Services][here]] and
[[https://gitlab.com/anemos-io/archlinux-ec2][here]] - only half work and not
194 maintained, but it's a start. I'm not even sure if I should stick with
195 arch or cave and use Ubuntu or AWS Linux. We can serve the static
196 services with little cost, the only big spender will be the heptapod
197 instance which requires a larger instance and some workers.
199 We'll try to keep the cost at or around $30/month.
203 :ID: 836b5ca7-6c6b-4d57-8b44-3bb4a8f38a38 207 :ID: fc9a94e1-91c5-4915-90b8-73218fa3b8bc 210 - State "TODO" from [2023-04-07 Fri 23:24] 220 :ID: c0613a13-7ccb-4af9-b47e-e14a41c782c2 223 - State "TODO" from "TODO" [2023-04-07 Fri 23:22] 228 :ID: 6bd29c37-4b23-4a05-b3a4-72b6ee712f27 231 - State "TODO" from [2023-04-07 Fri 23:33] 235 :ID: 3bfa733e-d777-4e82-b108-93ba74c4dc08 239 :ID: 0f3b6182-1dd1-4e65-8b1c-4630a46fff4c 243 :ID: e9fc19d5-daea-40a5-a0f2-882b925e58f2 247 :ID: c2e62af0-0397-4560-a283-b33d12fbd966 251 :ID: d2466880-1721-4386-b385-ded06d5b15cf 255 :ID: fb722462-9e11-40c1-a5d7-30698dba7e20 259 :ID: 663cd894-0a5f-4352-acf1-4528b9efeeaf 263 :ID: b4264fa2-6e96-4679-bd17-2428ba3489ee 266 - State "TODO" from [2023-04-07 Fri 23:52] 270 :ID: d9661e4a-0328-4864-878d-096f76ddaf55 273 - State "TODO" from [2023-04-07 Fri 23:52] 277 :ID: 501593d4-702a-40cb-8a8f-19edfe2f9f29 280 - State "TODO" from [2023-04-07 Fri 23:34] 284 :ID: 24700a26-b238-48dd-a24f-5539984b4fa6 287 - State "TODO" from [2023-04-07 Fri 23:51] 289 split out from rlib to separate package
290 - a purely OOP class library
293 :ID: fa1c9563-338b-4a56-bfc3-6f4983f2a732 296 - State "TODO" from [2023-04-07 Fri 23:34] 298 *** source categories 300 :ID: c43fdb5d-4f52-478e-9384-ec0e6e89cf09 302 - need a way of extracting metadata from a repo
303 - need ability to search and query libs/packages
304 - separate modules based on where they belong in our stack?
310 *** generic query language 312 :ID: 8647801a-d5ad-446d-8ae1-fd896bce129a 320 :ID: ff7c2876-e471-480b-b532-9921dbd3bc89 323 - Note taken on [2023-10-24 Tue 22:16] \\ 324 graph database, build on rocksdb 326 insidious Big Brother database.
327 - an application built with obj
332 :ID: 00e5fe6c-80b4-4172-a9cb-ce8adc40d3dc 335 - State "NOTE" from "TODO" [2024-08-18 Sun 18:46] 341 ** DRAFT dylib-skel-1 343 :ID: b4d1bc91-f344-45fd-becc-cb20f00a3a61 345 - State "DRAFT" from [2023-11-05 Sun 22:23]
348 :ID: 2e490c4b-344e-4790-9184-1c05ba675f15 350 Our core languages are
[[https://www.rust-lang.org/][Rust]] and
[[https://lisp-lang.org/][Lisp]] - this is the killer combo which will allow NAS-T
351 to rapidly develop high-quality software. As such, it's crucial that these two very
352 different languages (i.e. compilers) are able to interoperate seamlessly.
354 Some interop methods are easy to accomodate via the OS - such as IPC or data sharing,
355 but others are a bit more difficult.
357 In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something
358 that
/can/ be difficult, due to some complications with Rust and because this is not the
359 most popular software stack (yet ;). This is an experiment and may not make it to our
360 code-base, but it's definitely something worth adding to the toolbox in case we need it.
364 :ID: 985019fc-612a-44ab-b726-b9067432ad87 366 The level of interop we're after in this case is
[[https://en.wikipedia.org/wiki/Foreign_function_interface][FFI]].
368 Basically, calling Rust code from Lisp and vice-versa. There's an article about calling
369 Rust from Common Lisp
[[https://dev.to/veer66/calling-rust-from-common-lisp-45c5][here]] which shows the basics and serves as a great starting point
370 for those interested.
373 :ID: 2f71a3c1-0b14-46a6-9d8d-f6ec697729cc 375 The complication(s) with Rust I mentioned early is really just that
/it is not C/.
=C= 376 is old, i.e. well-supported with a stable ABI, making the process of creating bindings
377 for a C library a breeze in many languages.
379 For a Rust library we need to first appease the compiler, as explained in
[[https://doc.rust-lang.org/nomicon/ffi.html#calling-rust-code-from-c][this section]] 380 of the Rustonomicon. Among other things it involves changing the calling-convention of
381 functions with a type signature and editing the Cargo.toml file to produce a
382 C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used
387 :ID: 4ea79f68-55ec-4da3-a184-8343d49532b6 389 Using FFI involves some overhead. Check
[[https://github.com/dyu/ffi-overhead][here]] for an example benchmark across a few
390 languages. While building the NAS-T core, I'm very much aware of this, and will need a
391 few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular,
392 I'm concerned about crossing multiple language barriers (Rust
<->C
<->Lisp).
394 *** Rust -> C -> Lisp 396 :ID: a498276c-8525-4a43-aa40-4b05f76a29a9 400 :ID: 19f96ef7-af92-496e-9d42-70c4d4c85051 402 For starters, I'm going to assume we all have Rust (via
=rustup=) and Lisp (
=sbcl= only)
403 installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in
407 :ID: c929e0b6-b6f2-4383-9412-1610329ab28c 409 Create a new library crate. For this example we're focusing on a 'skeleton' for
410 /dynamic/ libraries only, so our experiment will be called
=dylib-skel= or
*dysk* for
412 src_sh[:exports code]{cargo init dysk --lib && cd dysk}
414 A
=src/lib.rs= will be generated for you. Go ahead and delete that. We're going to be
415 making our own
=lib.rs= file directly in the root directory (just to be cool).
417 The next step is to edit your
=Cargo.toml= file. Add these lines after the
=[package]= 418 section and before
=[dependencies]=:
419 #+begin_src conf-toml 421 crate-type = ["cdylib","rlib"]
428 This tells Rust to generate a shared C-compatible object with a
=.so= extension which we
429 can open using
[[https://man.archlinux.org/man/dlopen.3.en][dlopen]].
432 :ID: 256ac288-c5a0-473a-ab65-2d6503bd423c 436 :ID: fc476f64-6b68-417a-8540-ca23ce27fa25 438 Next, we want the
=cbindgen= program which we'll use to generate header files for
439 C/C++. This step isn't necessary at all, we just want it for further experimentation.
441 src_sh[:exports code]{cargo install --force cbindgen}
443 We append the
=cbindgen= crate as a
/build dependency/ to our
=Cargo.toml= like so:
444 #+begin_src conf-toml 450 :ID: 111e27f7-0b9c-4eef-9117-f7c8ba3f511c 452 #+begin_src conf-toml :tangle cbindgen.toml 454 autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */"
455 include_version = true
458 after_includes = "#define DYSK_VERSION \"0.1.0\""
462 documentation_style = "c99"
463 usize_is_size_t = true
469 :ID: 9fc271b2-9acb-4f4b-aa61-82d60d2ddb9e 471 #+begin_src rust :tangle build.rs 472 fn main() -> Result<(), cbindgen::Error> {
473 if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) {
474 b.write_to_file("dysk.h"); Ok(())}
475 else { panic!("failed to generate dysk.h from cbindgen.toml") } }
479 :ID: 6b524921-2ae0-43f0-bb85-d9955b0e689c 481 #+begin_src rust :tangle lib.rs 482 //! lib.rs --- dysk library
483 use std::ffi::{c_char, c_int, CString};
485 pub extern "C" fn dysk_hello() -> *const c_char {
486 CString::new("hello from rust").unwrap().into_raw()}
488 pub extern "C" fn dysk_plus(a:c_int,b:c_int) -> c_int {a+b}
490 pub extern "C" fn dysk_plus1(n:c_int) -> c_int {n+1}
494 :ID: cc7c6538-33a6-40c6-94ef-2a9c259c975a 496 #+begin_src rust :tangle test.rs 497 //! test.rs --- dysk test
498 fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::dysk_plus1(2 as core::ffi::c_int);}}
502 :ID: 337a24d1-f305-4e1a-9052-47a53591cb2f 505 cargo build --release
509 :ID: a4813269-92fb-4f52-aef0-3a36dce3cf69 511 #+begin_src lisp :tangle dysk.lisp 512 (load-shared-object #P"target/release/libdysk.so")
513 (define-alien-routine dysk-hello c-string)
514 (define-alien-routine dysk-plus int (a int) (b int))
515 (define-alien-routine dysk-plus1 int (n int))
516 (dysk-hello) ;; => "hello from rust"
520 :ID: 1a8ca441-f290-46c7-b979-1e7e0d1d063b 523 time target/release/dysk-test
525 #+begin_src lisp :tangle test.lisp 526 (time (dotimes (_ 500000000) (dysk-plus1 2)))
531 :ID: c5f4f25c-071b-4a2d-85bc-08676eacce5d 534 (defmethod cl-dot:graph-object-node ((graph (eql 'example)) (object cons))
535 (make-instance 'cl-dot:node
536 :attributes '(:label "cell \\N"
538 (defmethod cl-dot:graph-object-points-to ((graph (eql 'example)) (object cons))
540 (make-instance 'cl-dot:attributed
542 :attributes '(:weight 3))))
544 (defmethod cl-dot:graph-object-node ((graph (eql 'example)) (object symbol))
545 (make-instance 'cl-dot:node
546 :attributes `(:label ,object
550 :fillcolor "#ccccff")))
551 (let* ((data '(a b c #1=(b z) c d #1#))
552 (dgraph (cl-dot:generate-graph-from-roots 'example (list data)
553 '(:rankdir "LR" :layout "twopi" :labelloc "t"))))
554 (cl-dot:dot-graph dgraph "test-lr.svg" :format #+nil :x11 :svg))
561 (dgraph (cl-dot:generate-graph-from-roots 'example (list data)
563 (cl-dot:print-graph dgraph))
569 :ID: 7fb40057-05aa-445b-bc75-0b1b6ca2c994 571 need a way of indexing, referring to, and annotating objects such as
572 URLs, docs, articles, source files, etc.
574 What is the best way to get this done?
576 ** doc best practices 578 :ID: ba4a3074-f29d-4f81-b4ba-d6a523b3f291 580 https://rust-lang.github.io/api-guidelines/documentation.html
582 also: https:
//lisp-lang.org/style-guide/ 586 :ID: 7845b0fb-b5d3-4488-bdba-4876d9c8816e 588 If you've met me in the past decade, you probably know that I am
589 extremely passionate about computers. Let me first explain why.
591 On the most basic level computers are little (or big) machines that
592 can be programmed to do things, or
/compute/ if we're being
595 They host and provide access to the Internet, which is a pretty big
596 thing, but they do little things too like unlock your car door and
597 tell your microwave to beep at you. They solve problems. Big or small.
599 They're also
/everywhere/ - which can be scary to think about, but
600 ultimately helps propel us into the future.
602 There's something pretty cool about that - when you look at the
603 essence of computation. There are endless quantities of these machines
604 which follow the same basic rules and can be used to solve
/real/ 609 :ID: b17c853e-1728-4bd3-94c5-82e5b33d7398 611 Now, let us consider the
/programmer/. They have power.
/real/ 612 power. They understand the language of computers, can whisper to them
613 in various dialects. It can be intimidating to witness until you
614 realize how often the programmer says the wrong thing - a bug.
616 In reality, the programmer has a symbiotic relationship with
617 computers. Good programmers understand this relationship well.
620 One day after I got my first job at a software company, I remember
621 being on an all-hands meeting due to a client service outage. We had
622 some management, our lead devs, product team, and one curious looking
623 man who happened to be our lead IT consultant who had just joined. He
624 was sitting up on a hotel bed, shirtless, vaping an e-cig, typing
625 away in what I can only imagine was a shell prompt.
627 After several minutes he took a swig from a bottle of Coke and said
628 "Node 6 is sick." then a few seconds later our services were
629 restored. For the next hour on the call he explained what happened and
630 why, but that particular phrase always stuck with me. He didn't say
631 Node 6 was down, or had an expired cert - his diagnosis was that /it/
635 The more you work closely with computers, the more you start to think
636 of them this way. You don't start screaming when the computer does the
637 wrong thing, you figure out what's wrong and learn from it. With
638 experience, you start to understand the different behaviors of the
639 machines you work with. I like to call this
/Machine Empathy/.
643 :ID: a297e5f9-a875-4512-b126-9a2b3e75c1d8 645 I already mentioned bugs - I write plenty of those, but usually I try
646 to write
/programs/. Programs to me are like poetry. I like to think
647 they are for the computer too.
649 Just like computers,
/computer programs/ come in different shapes and
650 sizes but in basic terms they are sets of instructions used to control
653 You can write programs to do anything - when I first started, my
654 programs made music. The program was a means to an end. Over time, I
655 started to see the program as something much more. I saw it as the
658 [fn:1] ... perform computations
662 :ID: 80d800de-d305-4b95-a28c-a6f19d6f7a11 664 Something that is missing from many organizations big or large, is an
665 effective way to store and access information, even about their own
668 It can be difficult problem to solve - usually there's the official
669 one, say Microsoft Sharepoint and then the list of unofficial sources
670 which becomes tribal corporate hacker knowledge. Maybe the unofficial
671 ones are more current, or are annotated nicely, but their very
672 existence breaks the system. There's no longer a single source of
675 My priority in this department is writing services which process and
676 store information from a variety of sources in a distributed knowledge
677 graph. The graph can later be queried to access information on-demand.
679 My idea of infrastructure is in fact to build my own Cloud. Needless
680 to say I don't have an O365 subscription, and wherever possible I'll
681 be relying on hardware I have physical access to. I'm not opposed to
682 cloud services at large but based on principle I like to think we
683 shouldn't be built on them.
685 ** https://cal-coop.gitlab.io/utena/utena-specification/main.pdf 687 :ID: 5c3aac3a-67e1-4867-9ac4-b58a3247f9e0 689 from the author of cl-decentralise2. draft specification of a
690 /Maximalist/ Computing System.
694 :ID: 28738250-c34e-45e2-8793-f8245472d3a3 696 https:
//github.com/awesomedata/awesome-public-datasets 697 https://docs.openml.org/Datasets/ 698 https:
//wiki.pathmind.com/open-datasets 702 :ID: 28581a8e-ae05-45de-9744-24f447668418 705 sb-sys:*runtime-dlhandle* 706 sb-fasl:+fasl-file-version+ 707 sb-fasl:+backend-fasl-file-implementation+ 708 sb-debug:print-backtrace 709 sb-debug:map-backtrace 710 sb-pretty:pprint-dispatch-table 713 sb-walker:define-walker-template 714 sb-walker:macroexpand-all 717 sb-kernel:*eval-calls* 718 sb-kernel:*gc-pin-code-pages* 719 sb-kernel:*restart-clusters* 720 sb-kernel:*save-lisp-clobbered-globals* 721 sb-kernel:*top-level-form-p* 722 sb-kernel:*universal-fun-type* 723 sb-kernel:*universal-type* 724 sb-kernel:*wild-type* 725 sb-kernel:+simd-pack-element-types+ 727 (sb-vm:boxed-context-register) 728 (sb-vm:c-find-heap->arena) 729 (sb-vm:copy-number-to-heap) 730 (sb-vm:dump-arena-objects) 733 (sb-vm:show-heap->arena) 734 (sb-vm:with/without-arena) 735 (sb-cltl2:{augment-environment,compiler-let,define-declaration,parse-macro}) 736 (sb-cltl2:{declaration-information, variable-information, function-information}) 744 ** [[https://github.com/sigmf/SigMF][SigMF]]
746 :ID: f21a6477-6a3f-48f7-8a9e-43f5cf65286c 749 Sharing sets of recorded signal data is an important part of science
750 and engineering. It enables multiple parties to collaborate, is often
751 a necessary part of reproducing scientific results (a requirement of
752 scientific rigor), and enables sharing data with those who do not have
753 direct access to the equipment required to capture it.
755 Unfortunately, these datasets have historically not been very
756 portable, and there is not an agreed upon method of sharing metadata
757 descriptions of the recorded data itself. This is the problem that
760 By providing a standard way to describe data recordings, SigMF
761 facilitates the sharing of data, prevents the "bitrot" of datasets
762 wherein details of the capture are lost over time, and makes it
763 possible for different tools to operate on the same dataset, thus
764 enabling data portability between tools and workflows.
767 the-spec: https:
//github.com/sigmf/SigMF/blob/sigmf-v1.x/sigmf-spec.md 768 ** [[https://www.libvolk.org/][LibVOLK]]
770 :ID: ae34135c-488e-4aff-b795-a4d0d558f22b 772 Vector-Optimized Library of Kernels (simd)
773 ** [[https://docs.kernel.org/fb/framebuffer.html][/dev/fb*]] 775 :ID: 7819a970-1f37-40a6-985f-905b25a08a4a 777 framebuffers, used by fbgrab/fbcat program
778 ** [[https://docs.kernel.org/block/ublk.html][ublk]] 780 :ID: 5ccaed39-7055-4a51-ab62-769a73425270 782 https://github.com/ming1/ubdsrv
785 make problems smaller.
789 - doesn't need mentioning more and more
791 ** TODO taobench demo 793 :ID: 81ac81c3-f183-4345-aaf3-ef7ef0434947 796 - State "TODO" from [2024-01-21 Sun 00:32] 798 https:
//github.com/audreyccheng/taobench - shouldn't have missed this :) 799 obviously we need to implement this using core -- in demo/bench/tao? 800 ** TODO clap completion for nushell 802 :ID: a4b35281-83f7-4244-93e9-20e57ba31eb3 805 - State "TODO" from [2024-01-20 Sat 23:23] 807 https://github.com/clap-rs/clap/tree/master/clap_complete_nushell 808 ** Dataframe scripting 810 :ID: 8dc7339e-71c6-4a60-b285-c0d56d1cbc24 812 https://studioterabyte.nl/en/blog/polars-vs-pandas 813 nushell supports DFs, polars underneath? 814 https://www.nushell.sh/book/cheat_sheet.html
821 :ID: d5131c54-9082-4424-91dd-70f17561bbf7 825 :ID: 2b0f67de-1217-48a6-b5b7-a85d3ca15926 827 - [[https://cloud.google.com/free/docs/free-cloud-features][Free Cloud Features]] 828 + 90-day $300 credits
829 + e2-micro - free hours worth 1 instance/month
833 :ID: 0fd0dedd-59b0-4b1a-9241-f65699e20c46 838 :ID: 8315a33f-dd22-4374-979a-9cccd5b5b6a9 843 :ID: 3f1e1116-c123-4fdf-a336-0d9b50c3dcc7 845 - [[https://www.oracle.com/cloud/free/?intcmp=ohp052322ocift][OCI Free Tier]] 846 + always free: 2 x oracle autonomous DB
847 + 2 x AMD Compute VMs
848 + up to 4 x ARM Ampere A1 with 3k/cpu/hr and 18k/gb/h per month
849 + block/object/archive storage
850 + 30-day $300 credits
852 ** trash as block device 854 :ID: 4452b84f-f3a5-44f1-b60a-4efa6413cbb9 857 - State "NOTE" from [2024-01-29 Mon 20:53] 858 - State "NOTE" from [2024-01-29 Mon 20:53] 860 in nushell there is option for rm command to always use 'trash' -
861 AFAIK the current approach is via a service (trashd).
863 An interesting experiment would be to designate a block device as
864 'trash' - may be possible to remove reliance on a service
866 may be an opportunity for ublk driver to shine - instead of /dev/null
867 piping we need a driver for streaming a file to /dev/trash
870 :ID: aa10a1d7-f1e7-4b92-b91a-56c234d52204 873 - State "NOTE" from [2024-01-29 Mon 16:28] 875 - mostly x86_64 machines - currently 2 AWS EC2 instances, some podman containers, and our home beowulf server:
878 - mid-size tower enclosed (Linux/Windows)
880 - Intel Core i7-6700K
883 - NVIDIA GeForce GTX 1060
886 - Samsung SSD 850: 232.9GB
887 - Samsung SSD 850: 465.76GB
888 - ST2000DM001-1ER1: 1.82TB
889 - WDC WD80EAZZ-00B: 7.28TB
890 - PSSD T7 Shield: 3.64TB
891 - My Passport 0820: 1.36TB
893 - 16GB (2*8) [64GB max]
896 - MacBook Pro 2019 (MacOS/Darwin)
912 - 64-bit 6-core 4x ARM Cortex A53 + 2x ARM Cortex A72
918 - Raspberry Pi 4 Model B
920 - Cortex-A72 (ARM v8) 64-bit SoC
928 :ID: 87d0f5d4-c6fe-4e21-911e-2639a24e89f9 932 let ms = '1trc/measurements-0.parquet'
934 | dfr group-by station
936 (dfr col measure | dfr min | dfr as "min")
937 (dfr col measure | dfr max | dfr as "max")
938 (dfr col measure | dfr sum | dfr as "sum")
939 (dfr col measure | dfr count | dfr as "count")
945 :ID: b4b29989-28f2-48db-ad84-cd8d08eeefdf 948 - State "NOTE" from [2024-02-18 Sun 11:55] 950 In the past few months there has been drama regarding Wayland vs X. It
951 seems to be on everyone's minds after Artem's freakout issue and the
952 follow up YT vids/comments.
954 I admit that it made me reconsider the fitness of WL as a whole -
955 there was a github gist that made some scathing arguments against it.
957 It's an odd debate though. I think there are many misunderstandings.
959 So first off, if we look at the homepage
960 https:
//wayland.freedesktop.org/, Wayland claims it is a replacement
961 for X11. It now has
/manifest destiny/, which in my opinion is a great
964 X-pros seem to agree that Wayland has
/manifest destiny/ - like if you
965 are building softwares that look remotely like a window system, it's a
966 successor to X. That's the model of doing things and there's no way
969 The disagreement starts with how this destiny - of an X2 - should be
970 fulfilled. X-pros want a fork of X, but it's too late for
971 that. WL-pros want X to run on top of Wayland compositor:
972 https://wayland.freedesktop.org/xserver.html.
974 Xwayland is a problem for me. From the project description: 'if we're
975 migrating away from X, it makes sense to have a good backwards
976 compatibility story.' Full disclosure: I have never done significant
977 work on Xwayland, so perhaps my opinion is unwarranted. But I have no
978 intention of attempting to maintain a computer system that uses
979 Wayland and X clients at the same time.
981 To me, X is ol' reliable. Every distro has first-class X support, and
982 it runs on most systems with very little user intervention. Where it
983 doesn't, there is 20+ years of dev history and battle-tested
984 workarounds for you to find your solution in.
986 Wayland is the new kid on the block, born just in 2008. It's a fresh
987 start to one of the most difficult challenges in software - window
988 systems. A re-write would be pointless though, and so the real
989 value-add is in design. Wayland is designed as a protocol and
990 collection of libraries which are implemented in your own
991 compositor. Coming from Lisp - with ANSI Common Lisp and SRFIs, this
992 feels right even if the implementation is something very different
993 (compositor vs compiler).
995 With X, it is assumed to be much harder to write an equivalent
996 'compositor'. Here's the thing though - with a significantly complex X
997 client implementation, it is
/impossible/ to replicate in WL. This is
998 really the crux of Artemi's argument in his issue. He asked for a 1:1
999 equivalent X/WL comparison when no such thing exists, and in my
1000 opinion it is a waste of time.
1002 The WL core team is fully aware of this dichotomy, but also that this
1003 is in no way a problem or weakness in either system. It means they're
1004 different systems, goddammit.
1006 If it was up to me, Xwayland wouldn't exist. I understand why it does,
1007 and that it does make things easier for developers who need to support
1008 both, and users who have multiple apps with multiple windowing
1009 requirements. It's a bandaid though, and one that is particularly
1010 dangerous because it re-enforces the idea that Wayland is just X2 and
1011 that they're fully compatible.
1013 What interests me in the Wayland world right now is the idea of a
1014 small, modular, full-stack Wayland compositor API. There are several
1015 'kiosk' based compositors for single applications (cage), but these
1016 aren't complete solutions. It is possible to get much closer to the
1017 metal, and that's where I want to be so that I can build my own APIs
1018 on top - I don't want to live on top of X, and I certainly don't want
1019 to live on top of X on top of WL. I want a
/pure/ solution that hides
1020 as little as possible, exposing the interesting bits.
1022 ** TODO collect more data 1024 :ID: 69ce5703-d3fe-4df7-abf4-dc93a08ebc12 1027 - State "TODO" from [2024-03-01 Fri 15:27] 1029 https:
//www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/
1030 weather - music - etc
1032 ** On blocks and devices 1034 :ID: c8304ae3-415d-4ccf-8f4c-1f3f6a555e5b 1037 - State "NOTE" from [2024-03-02 Sat 21:30] 1039 [[https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/dev.html][/dev]] 1040 In Linux, everything is a file.
1042 /dev/ contains special device files - usually block or character
1045 major, minor = category, device
1048 mknod - create special device files
1050 [[https://www.redhat.com/en/blog/linux-block-devices-hints-debugging-and-new-developments][redhat hints]] 1053 dd if=/dev/zero of=myfile bs=1M count=32
1054 losetup --show -f myfile
1056 losetup -d /dev/loop0 #teardown
1060 echo "sup dude" > /dev/loop0
1061 dd if=/dev/loop0 -bs=1
1062 dd if=/dev/nvme0 of=/dev/null progress=true
1064 hdparm -T /dev/nvme0
1068 modprobe scsi_debug add_host=5 max_luns=10 num_tgts=2 dev_size_mb=16
1071 sparsefiles: create with C, dd, or truncate
1073 #+begin_src shell :results output 1079 Usage: truncate OPTION... FILE...
1080 Shrink or extend the size of each FILE to the specified size
1082 A FILE argument that does not exist is created.
1084 If a FILE is larger than the specified size, the extra data is lost.
1085 If a FILE is shorter, it is extended and the sparse extended part (hole)
1086 reads as zero bytes.
1088 Mandatory arguments to long options are mandatory for short options too.
1089 -c, --no-create do not create any files
1090 -o, --io-blocks treat SIZE as number of IO blocks instead of bytes
1091 -r, --reference=RFILE base size on RFILE
1092 -s, --size=SIZE set or adjust the file size by SIZE bytes
1093 --help display this help and exit
1094 --version output version information and exit
1096 The SIZE argument is an integer and optional unit (example: 10K is 10*1024).
1097 Units are K,M,G,T,P,E,Z,Y,R,Q (powers of 1024) or KB,MB,... (powers of 1000).
1098 Binary prefixes can be used, too: KiB=K, MiB=M, and so on.
1100 SIZE may also be prefixed by one of the following modifying characters:
1101 '+' extend by, '-' reduce by, '<' at most, '>' at least,
1102 '/' round down to multiple of, '%' round up to multiple of.
1104 GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
1105 Full documentation <https://www.gnu.org/software/coreutils/truncate>
1106 or available locally via: info '(coreutils) truncate invocation'
1109 test mkfs.btrfs on 10T dummy block device
1111 dd if=/dev/zero of=/tmp/bb1 bs=1 count=1 seek=10T
1113 losetup --show -f /tmp/bb1
1114 mkfs.btrfs /dev/loop0
1119 iostat # pacman -S sysstat
1120 blktrace # paru -S blktrace
1121 iotop # pacman -S iotop
1124 bcc/ trace: Who/which process is executing specific functions against
1127 bcc/biosnoop: Which process is accessing the block device, how many
1128 bytes are accessed, which latency for answering the requests?
1130 at the kernel level besides BPF we got kmods and DKMS,
1132 compression/de-duplication can be done via VDO kernel mod
1134 https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
1136 ** save-lisp-and-respawn 1138 :ID: e58a96e7-7b88-4e4a-b958-3000603d8baa 1141 - State "NOTE" from [2024-03-02 Sat 22:57] 1149 :ID: 2efa5e4a-f063-49bb-9e61-6917eb115c1a 1152 - State "NOTE" from [2024-03-03 Sun 16:35] 1155 - openlog syslog closelog
1156 - levels: emerg alert crit err warning notice info debug
1159 ** RESEARCH [[https://github.com/guicho271828/sbcl-wiki/wiki/][sbcl-wiki]] 1161 :ID: 33bc81c1-c5fa-4518-9456-58fd67f301ff 1164 - State "RESEARCH" from [2024-03-13 Wed 21:49] 1168 :ID: 2c012572-ea16-408c-b741-d1f6cd0c7961 1172 :ID: 8d015e5e-7ad3-44c3-88cd-81e293a887e0 1177 :ID: d30aec42-a9a1-425b-87ca-1e40db0e69de 1180 - State "NOTE" from [2024-02-04 Sun 20:40] 1182 [[https://github.com/facebook/rocksdb/wiki/Benchmarking-tools][RocksDB benchmarking tools]] 1186 :ID: 76ae24f5-46e8-4b91-8991-41245383d337 1189 - State "NOTE" from [2024-01-25 Thu 22:39] 1191 - API root: https:
//packy.compiler.company 1192 - source packs: https://vc.compiler.company/packy 1195 :ID: 2d6aa6fb-a34c-4ccd-99d3-4f98d33bb2f9 1199 :ID: 8017b6c7-3a04-4e52-b3d6-afc68cbce139 1203 :ID: 2722b7dc-9546-4d94-82b0-abfc02647b4d 1205 Primary data type of the library - typically represents a compressed 1206 archive, metadata, and ops. 1209 :ID: 9ec5d682-da8e-4d02-b2fc-a56a8f209c22 1211 Collection data type, usually contains a set of packs with metadata. 1214 :ID: 8cf2658a-92d5-41ab-8630-4f4e6ddb140d 1216 Represents a Packy instance bound to a UDP socket 1217 ***** PackyEndpointConfig 1219 :ID: 421b8aaa-66e1-4883-937d-fc6705146998 1221 Global endpoint configuration object 1222 ***** PackyClientConfig 1224 :ID: c1899710-034e-4ed6-959a-1eb2df72e5f9 1226 Configuration for outgoing packy connections on an endpoint 1227 ***** PackyServerConfig 1229 :ID: 955e8ce9-ff7e-4f42-9d25-3aeb9846528b 1231 Configuration for incoming packy connection son an endpoint 1232 ***** PackyConnection 1234 :ID: 7075e0c8-65b3-4fbf-9a6b-f27017276a72 1236 Packy connection object 1239 :ID: ea92ad07-d12c-4ee8-a3d4-0955f0af3921 1243 :ID: 6c0fdd7e-bc0a-4217-b315-72fe7766e4df 1247 :ID: 1358e1c5-b215-48eb-9e1b-4116cc32b2ae 1251 :ID: b8298207-4d92-4628-8687-9f4e20c8cfec 1255 :ID: a6fb4fb6-b188-4033-8291-107cf1f00e6d 1259 :ID: 1fd91c8b-ab90-4db0-b74e-1c2b9ae2e440 1263 :ID: 75c0db2e-37cf-4f33-95bd-3f5c95b45c49 1267 :ID: 60de92f4-b4a6-4cae-95bc-174e2036e57b 1271 :ID: e379b71e-8a84-4547-a694-90cad3b7952e 1275 :ID: 73ea9478-5251-40c3-8331-cf042d0b9bf1 1277 ****** start_packy_server 1279 :ID: c8a9bac3-971a-45a7-988f-92db19ef6601 1281 ****** stop_packy_server 1283 :ID: e3122bf1-f770-46a7-a36b-2df9c48bc233 1285 ****** start_packy_registry 1287 :ID: b6d214f2-d6de-4d46-b2c2-c51ff0f3b3e6 1291 :ID: 0e846cbb-5cb4-496f-8100-beae6e66972e 1293 ****** register_pack 1295 :ID: 764642ad-a33b-415b-9c32-8a729a7cd62d 1297 ****** register_user 1299 :ID: 2b92f584-d769-4266-940e-cfbd466d4e62 1301 ****** register_bundle 1303 :ID: 390cb42b-5170-4175-9ccc-b990842356a0 1306 ** TBD investigate alieneval for phash opps 1308 :ID: 21ab0a6e-cb66-4c5e-9b10-4ac2340ce75d 1311 - State "TBD" from [2024-03-25 Mon 18:56] 1316 :ID: 439a4945-de2f-4985-ba6c-fe6421aa9ad9 1318 The backend services are written in Rust and controlled by a simple 1319 messaging protocol. Services provide common runtime capabilities known 1320 as the /service protocol/ but are specialized on a unique
/service 1321 type/ which may in turn register their own
/custom protocols/ (via
1324 Services are capable of dispatching data directly to clients, or
1325 storing data in the
/database/ (sqlite, postgres, mysql).
1327 The frontend clients are pre-dominantly written in Common Lisp and
1328 come in many shapes and sizes. There is a cli-client, web-client
1329 (CLOG), docker-client (archlinux, stumpwm, McCLIM), and native-client
1330 which also compiles to WASM (slint-rs).
1334 :ID: 65140d2c-5d59-48cd-946c-01c380a37f08 1338 :ID: ad0652ca-4abf-4d27-8897-a61a13a6e61d 1340 - *install dependencies* 1344 - *make executables* \\ 1345 Simply run
=make build=. Read the
~makefile~ and change the options
1347 - MODE :: Mode (debug, release)
1348 - LISP :: Lisp (sbcl, cmucl, ccl)
1349 - CFG :: Config (default.cfg)
1352 :ID: b2295442-ccb7-4a5c-8a9e-ea8502aad636 1359 :ID: 600be170-d032-4280-a7b5-c46e3c157206 1361 Configs can be specified in JSON, TOML, RON, or of course SEXP. See
1362 =default.cfg= for an example.
1365 :ID: 56a69ed7-2eb3-4490-95a1-072cee985df5 1367 The high-level user interface is presented as a multi-modal GUI
1368 application which adapts to the specific application
/instances/ 1372 :ID: 804a174b-534c-4a04-966a-b44c0a86fbd4 1374 This backend retrieves weather data using the NWS API.
1377 :ID: 4e7339bb-aaf3-4fab-90e4-05e13f24f881 1379 The 'Stocks' backend features a stock ticker with real-time analysis
1383 :ID: e3cf0add-6ded-4c05-a05d-7ef463b85b08 1385 This is a benchmark backend for testing the capabilities of our
1386 demo. It spins up some mock services and allows fine-grained control
1387 of input/throughput.
1390 :ID: a16a3807-0e06-4c8d-baf6-5dfa8f8f5585 1394 :ID: 3ae4754e-0c0f-4923-9e53-ae962b8638a1 1396 - consider tree-sitter parsing layout, use as a guide for developing a
1397 single syntax which expands to Rust or C.
1403 **** TODO rs-macroexpand 1405 :ID: f612975f-6bcc-4aab-b0d8-65b8ff0a0d0f 1412 **** TODO c-macroexpand 1414 :ID: 49f5c224-026d-4cdc-a326-1742e64c4279 1421 **** TODO slint-macroexpand 1423 :ID: 486beb12-179b-4acc-8507-cdc498edb628 1429 - slint-macroexpand-1
1430 **** TODO html (using who) 1432 :ID: 2e81a19c-874c-4fd7-be0f-f6cd62c660db 1434 *** TODO web templates 1436 :ID: 95f7d6ac-c010-4d1d-a47d-1693bb7d6eab 1438 create a basic static page in CL which will be used to host Slint UIs
1439 and other WASM doo-dads in a browser.
1442 :ID: fc1d5364-279c-4d5f-baea-ed2c110b63f0 1444 using clingon, decide on generic options and write it up
1447 :ID: 467e84e3-e72f-4764-8858-cc15fdc17e8a 1449 work on doc generation -- Rust and CL should be accounted for.
1452 :ID: 70aea01c-d01e-4ddf-aaf5-c86183a7e250 1454 We have none! need to make it more comfy - set up testing in all Rust
1455 crates and for the lisp systems.
1457 ** https://docs.gitlab.com/ee/administration/backup_restore/migrate_to_new_server.html 1459 :ID: 264b81f4-f953-4698-9c82-d8e20b6c31f9 1464 :ID: 635925f6-e1d8-47bc-a32f-28bca0f19d2d 1466 - make sure to apk add:
1476 :ID: 6e9c6bce-0513-4628-b6f9-5c234cccdaa3 1478 - How should such objects be represented within CORE?
1480 - already have alias
1481 - not sure about obj/otherwise, prob not
1483 ** Intelligent Design in Software 1485 :ID: 1397d8bb-4848-4d9a-9dcc-ee65bed81c9e 1487 - starting from a space where there are no external influences - a biome
1488 - answer questions regarding the nature of the software and its capabilities
1489 - incrementally adjust inter-dependencies
1491 - protect the biome at all costs
1492 - focus on composition
1493 - build applications
1494 - re-integrate lessons learned
1496 ** keys.compiler.company 1498 :ID: 86c21790-72b5-4ea1-ba49-f7be3f65a8cc 1501 - State "NOTE" from [2024-08-16 Fri 22:00] 1503 - public openpgp server
1504 - [[https://keys.compiler.company][keys.compiler.company]] 1505 - https:
//keys.openpgp.org/ 1506 - [[vc:packy/hagrid/][packy/hagrid]]