# HG changeset patch # User ellis # Date 1699242375 18000 # Node ID bd85a72319d85b4cc4d96df2459a6c500ec26271 # Parent 7895e9b829178ad5e0970a9ad6b058dec7544dc4 migrated nas-t docs diff -r 7895e9b82917 -r bd85a72319d8 nas-t/api.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/api.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,5 @@ +#+TITLE: api +* Overview +* Protocols +* Filesystems +* Objects diff -r 7895e9b82917 -r bd85a72319d8 nas-t/install.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/install.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,6 @@ +#+TITLE: install +* Overview +#+begin_src sh + make build + # make install +#+end_src diff -r 7895e9b82917 -r bd85a72319d8 nas-t/notes.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/notes.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,235 @@ +#+TITLE: notes +#+BIBLIOGRAPHY: refs.bib +* File Systems +** BTRFS +#+begin_quote +BTRFS is a Linux filesystem based on copy-on-write, allowing for +efficient snapshots and clones. + +It uses B-trees as its main on-disk data structure. The design goal is +to work well for many use cases and workloads. To this end, much +effort has been directed to maintaining even performance as the +filesystem ages, rather than trying to support a particular narrow +benchmark use-case. + +Linux filesystems are installed on smartphones as well as enterprise +servers. This entails challenges on many different fronts. + +- Scalability :: The filesystem must scale in many dimensions: disk + space, memory, and CPUs. + +- Data integrity :: Losing data is not an option, and much effort is + expended to safeguard the content. This includes checksums, metadata + duplication, and RAID support built into the filesystem. + +- Disk diversity :: The system should work well with SSDs and hard + disks. It is also expected to be able to use an array of different + sized disks, which poses challenges to the RAID and striping + mechanisms. +#+end_quote +-- [cite/t/f:@btrfs] +*** [2023-08-08 Tue] btrfs performance speculation :: + - [[https://www.percona.com/blog/taking-a-look-at-btrfs-for-mysql/]] + - zfs outperforms immensely, but potential misconfiguration on btrfs side (virt+cow + still enabled?) + - https://www.ctrl.blog/entry/btrfs-vs-ext4-performance.html + - see the follow up comment on this post + - https://www.reddit.com/r/archlinux/comments/o2gc42/is_the_performance_hit_of_btrfs_serious_is_it/ + #+begin_quote + I’m the author of OP’s first link. I use BtrFS today. I often shift lots of + de-duplicatable data around, and benefit greatly from file cloning. The data is actually + the same data that caused the slow performance in the article. BtrFS and file cloning + now performs this task quicker than a traditional file system. (Hm. It’s time for a + follow-up article.) + + In a laptop with one drive: it doesn’t matter too much unless you do work that benefit + from file cloning or snapshots. This will likely require you to adjust your tooling and + workflow. I’ve had to rewrite the software I use every day to make it take advantage of + the capabilities of a more modern file system. You won’t benefit much from the data + recovery and redundancy features unless you’ve got two storage drives in your laptop and + can setup redundant data copies. + + on similar hardware to mine? + + It’s not a question about your hardware as much as how you use it. The bad performance I + documented was related to lots and lots of simultaneous random reads and writes. This + might not be representative of how you use your computer. + #+end_quote + - https://dl.acm.org/doi/fullHtml/10.1145/3386362 + - this is about distributed file systems (in this case Ceph) - they argue against + basing DFS on ondisk-format filesystems (XFS ext4) - developed BlueStore as + backend, which runs directly on raw storage hardware. + - this is a good approach, but expensive (2 years in development) and risky + - better approach is to take advantage of a powerful enough existing ondisk-FS + format and pair it with supporting modules which abstract away the 'distributed' + mechanics. + - the strategy presented here is critical for enterprise-grade hardware where the + ondisk filesystem becomes the bottleneck that you're looking to optimize + - https://lore.kernel.org/lkml/cover.1676908729.git.dsterba@suse.com/ + - linux 6.3 patch by David Sterba [2023-02-20 Mon] + - btrfs continues to show improvements in the linux kernel, ironing out the kinks + - makes it hard to compare benchmarks tho :/ +*** MacOS support +- see this WIP k-ext for macos: [[https://github.com/relalis/macos-btrfs][macos-btrfs]] + - maybe we can help out with the VFS/mount support +*** on-disk format +- [[https://btrfs.readthedocs.io/en/latest/dev/On-disk-format.html][on-disk-format]] +- 'btrfs consists entirely of several trees. the trees use copy-on-write.' +- trees are stored in nodes which belong to a level in the b-tree structure. +- internal nodes (inodes) contain refs to other inodes on the /next/ level OR + - to leaf nodes then the level reaches 0. +- leaf nodes contain various types depending on the tree. +- basic structures + - 0:8 uint = objectid, each tree has its own set of object IDs + - 8:1 uint = item type + - 9:8 uint = offset, depends on type. + - little-endian + - fields are unsigned + - *superblock* + - primary superblock is located at 0x10000 (64KiB) + - Mirror copies of the superblock are located at physical addresses 0x4000000 (64 + MiB) and 0x4000000000 (256GiB), if valid. copies are updated simultaneously. + - during mount only the first super block at 0x10000 is read, error causes mount to + fail. + - BTRFS onls recognizes disks with a valid 0x10000 superblock. + - *header* + - stored at the start of every inode + - data following it depends on whether it is an internal or leaf node. + - *inode* + - node header followed by a number of key pointers + - 0:11 key + - 11:8 uint = block number + - 19:8 uint = generation + - *lnode* + - leaf nodes contain header followed by key pointers + - 0:11 key + - 11:4 uint = data offset relative to end of header(65) + - 15:4 uint = data size +- objects + - ROOT_TREE + - holds ROOT_ITEMs, ROOT_REFs, and ROOT_BACKREFs for every tree other than itself. + - used to find the other trees and to determine the subvol structure. + - holds items for the 'root tree directory'. laddr is store in the superblock + - objectIDs + - free ids: BTRFS_FIRST_FREE_OBJECTID=256ULL:BTRFS_LAST_FREE_OBJECTID=-256ULL + - otherwise used for internal use +*** send-stream format +- [[https://btrfs.readthedocs.io/en/latest/dev/dev-send-stream.html][send stream format]] +- Send stream format represents a linear sequence of commands describing actions to be + performed on the target filesystem (receive side), created on the source filesystem + (send side). +- The stream is currently used in two ways: to generate a stream representing a + standalone subvolume (full mode) or a difference between two snapshots of the same + subvolume (incremental mode). +- The stream can be generated using a set of other subvolumes to look for extent + references that could lead to a more efficient stream by transferring only the + references and not full data. +- The stream format is abstracted from on-disk structures (though it may share some + BTRFS specifics), the stream instructions could be generated by other means than the + send ioctl. +- it's a checksum+TLV +- header: u32len,u16cmd,u32crc32c +- data: type,length,raw data +- the v2 protocol supports the encoded commands +- the commands are kinda clunky - need to MKFIL/MKDIR then RENAM to create +*** [2023-08-09 Wed] ioctls +- magic#: 0x94 + - https://docs.kernel.org/userspace-api/ioctl/ioctl-number.html + - Btrfs filesystem some lifted to vfs/generic + - fs/btrfs/ioctl.h and linux/fs.h +** ZFS +-- [cite/t/f:@zfs] + +- core component of TrueNAS software +** TMPFS +-- [cite/t/f:@tmpfs] +- in-mem FS +** EXT4 +-- [cite/t/f:@ext4] +** XFS +-- [cite/t/f:@xfs] +-- [cite/t/f:@xfs-scalability] +* Storage Mediums +** HDD +-- [cite/t/f:@hd-failure-ml] +** SSD +-- [cite/t/f:@smart-ssd-qp] +-- [cite/t/f:@ssd-perf-opt] + +** Flash +-- [cite/t/f:@flash-openssd-systems] +** NVMe +-- [cite/t/f:@nvme-ssd-ux] +-- [[https://nvmexpress.org/specifications/][specifications]] +*** ZNS +-- [cite/t/f:@zns-usenix] +#+begin_quote +Zoned Storage is an open source, standards-based initiative to enable data centers to +scale efficiently for the zettabyte storage capacity era. There are two technologies +behind Zoned Storage, Shingled Magnetic Recording (SMR) in ATA/SCSI HDDs and Zoned +Namespaces (ZNS) in NVMe SSDs. +#+end_quote +-- [[https://zonedstorage.io/][zonedstorage.io]] +-- $465 8tb 2.5"? [[https://www.serversupply.com/SSD/PCI-E/7.68TB/WESTERN%20DIGITAL/WUS4BB076D7P3E3_332270.htm][retail]] +** eMMC +-- [cite/t/f:@emmc-mobile-io] +* Linux +** syscalls +*** ioctl +- [[https://elixir.bootlin.com/linux/latest/source/Documentation/userspace-api/ioctl/ioctl-number.rst][ioctl-numbers]] +* Rust +** crates +*** nix +- [[https://crates.io/crates/nix][crates.io]] +*** memmap2 +- [[https://crates.io/crates/memmap2][crates.io]] +*** zstd +- [[https://crates.io/crates/zstd][crates.io]] +*** rocksdb +- [[https://crates.io/crates/rocksdb][crates.io]] +*** tokio :tokio: +- [[https://crates.io/crates/tokio][crates.io]] +*** tracing :tokio: +- [[https://crates.io/crates/tracing][crates.io]] +**** tracing-subscriber +- [[https://crates.io/crates/tracing-subscriber][crates.io]] +*** axum :tokio: +- [[https://crates.io/crates/axum][crates.io]] +*** tower :tokio: +- [[https://crates.io/crates/tower][crates.io]] +*** uuid +- [[https://crates.io/crates/uuid][crates.io]] +** unstable +*** lazy_cell +- [[https://github.com/rust-lang/rust/issues/109736][tracking-issue]] +*** {BTreeMap,BTreeSet}::extract_if +- [[https://github.com/rust-lang/rust/issues/70530][tracking-issue]] +* Lisp +** ASDF +- [[https://gitlab.common-lisp.net/asdf/asdf][gitlab.common-lisp.net]] +- [[https://asdf.common-lisp.dev/][common-lisp.dev]] +- [[https://github.com/fare/asdf/blob/master/doc/best_practices.md][best-practices]] +- includes UIOP +** Reference Projects +*** StumpWM +- [[https://github.com/stumpwm/stumpwm][github]] +*** Nyxt +- [[https://github.com/atlas-engineer/nyxt][github]] +*** Kons-9 +- [[https://github.com/kaveh808/kons-9][github]] +*** cl-torrents +- [[https://github.com/vindarel/cl-torrents][github]] +*** Mezzano +- [[https://github.com/froggey/Mezzano][github]] +*** yalo +- [[https://github.com/whily/yalo][github]] +*** cl-ledger +- [[https://github.com/ledger/cl-ledger][github]] +*** Lem +- [[https://github.com/lem-project/lem][github]] +*** kindista +- [[https://github.com/kindista/kindista][github]] +*** lisp-chat +- [[https://github.com/ryukinix/lisp-chat][github]] +* Refs +#+print_bibliography: diff -r 7895e9b82917 -r bd85a72319d8 nas-t/quickstart.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/quickstart.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,4 @@ +#+TITLE: quickstart +* Install +* Configure the daemon +* Connect a client diff -r 7895e9b82917 -r bd85a72319d8 nas-t/readme.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/readme.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,15 @@ +#+TITLE: index +#+DESCRIPTION: NAS-T Documentation + +Welcome to the documentation home of NAS-T. Choose one of the topics below to get +started. + +- [[file:quickstart.org][Quickstart]] :: get your private cloud now. + +- [[file:install.org][Install]] :: complete installation guide. + +- [[file:tutorial.org][Tutorial]] :: in-depth tutorial with detailed examples. + +- [[file:roadmap.org][Roadmap]] :: The NAS-T development roadmap. + +- [[file:api.org][API]] :: API Documentation for NAS-T hackers. diff -r 7895e9b82917 -r bd85a72319d8 nas-t/roadmap.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/roadmap.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,110 @@ +#+TITLE: roadmap +* 0.1.0 [0/3] :current: +** TODO MVP [1/6] +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:14] +:END: +*** DONE acquire domain +:LOGBOOK: +- State "DONE" from [2023-08-01 Tue 19:16] +:END: +- [X] nas-t.net +*** DRAFT nas-t.net :web:rust: +:LOGBOOK: +- State "DRAFT" from "TODO" [2023-08-23 Wed 22:21] +- State "TODO" from [2023-08-01 Tue 19:14] +:END: +A basic landing page with the NAS-T mission, links, and contact info +is needed. + +The site should be a web-app and have basic client-side request +capabilities for submitting inquiries/comments. +**** TODO client [0/3] :client: +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:23] +:END: +SPA +***** TODO styles :style: +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:19] +:END: +Put together some color palettes and sass to make our web components +pretty. Design in modular fashion. +***** TODO components +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:22] +:END: +- [X] index +- [X] about +- [X] contact + - [ ] >ty +- [X] page +- [X] app +- [ ] err +***** TODO hooks +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:22] +:END: +****** TODO mailme +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:30] +:END: +**** TODO server [0/2] :server: +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:22] +:END: +***** TODO mailme route +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:29] +:END: +- form submit + - email + - name + - message +- return status +***** TODO errors +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:32] +:END: +- basic scaffolding +- [ ] impl IntoResponse +*** OUTLINE daemon :design: +:LOGBOOK: +- State "OUTLINE" from "TODO" [2023-08-23 Wed 22:21] +- State "TODO" from [2023-08-01 Tue 19:33] +:END: +*** OUTLINE protocol :design: +:LOGBOOK: +- State "OUTLINE" from "TODO" [2023-08-23 Wed 22:21] +- State "TODO" from [2023-08-01 Tue 19:34] +:END: +*** TODO nas-t CLI client :design:impl: +:LOGBOOK: +- State "TODO" from [2023-08-01 Tue 19:34] +:END: +*** TBD prototype +:LOGBOOK: +- State "TBD" from "TODO" [2023-08-23 Wed 22:22] +- State "TODO" from [2023-08-01 Tue 19:36] +:END: +- proof-of-concept on x86-64 GNU/Linux Intel CPU (zor) +- run on LAN with remote clients +- show capabilities and limitations +- perform basic benchmarks and discuss results +** TBD pitch [0/0] +- marketing/community approach +- define rhetoric +- optimized for variety of audiences +** TBD demo [0/0] +- assess hardware options + - soc + - must run GNU/Linux (for now..) + - look locally, what can be re-purposed? + - compare performance,heat,cost + - storage + - define required I/O speeds + - define standard connector reqs based on soc + - optimize linux kernel/dist +- infra/releng infra/deploy +- no accessories, ssh access only (or display port based on soc) +* 0.2.0 [0/1] :next: diff -r 7895e9b82917 -r bd85a72319d8 nas-t/tutorial.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nas-t/tutorial.org Sun Nov 05 22:46:15 2023 -0500 @@ -0,0 +1,14 @@ +#+TITLE: tutorial +* Overview +** Introduction +** The Cloud +*** Services +**** Google +**** DropBox +**** Amazon +*** Protocols +**** HTTP +- gdrive,dropbox,s3 +**** RSYNC +**** FTP +**** WebDAV diff -r 7895e9b82917 -r bd85a72319d8 projects.org diff -r 7895e9b82917 -r bd85a72319d8 style.org --- a/style.org Sun Nov 05 21:44:43 2023 -0500 +++ b/style.org Sun Nov 05 22:46:15 2023 -0500 @@ -10,14 +10,35 @@ [fn:1] https://google.github.io/styleguide/ +* General Programming + +- indent-offset = 2 +- max-line-width = 80 +- page-length ~ 32 +- module-length ~ 512 +- file-size ~ 1024 + +- start every source-file with a header comment +- use outline headings to organize your program +- use ulang-comments as needed for referencing bugs, notes, etc. +- inline documentation should be declarative. your code explains your + comments for you. + * Common Lisp +- prefer symbol docs to comment docs + +** File templates * Rust +- don't make =mod.rs= files + - use the =foo.rs=, =foo/*= pattern instead +** File templates * Emacs Lisp - +** File templates * Org-mode - +** File templates * Shell - +** File templates * Python +** File templates