pages / Design notes
Design notes
on this page
The machine#
0x0000_0000 .. memory_size RAM (kernel Image at 0x0020_0000; DTB and initrd at the top)
0xE000_0000 .. 0xEFFF_FFFF PCI MMIO window (virtio BARs, 0x10000 per slot)
0xF000_0000 GICv2 distributor 0xF001_0000 GICv2 CPU interface
0xF010_0000 16550 UART (ttyS0) 0xF011_0000 PL031 RTC
0xF012_0000 VM generation ID (16 bytes, SPI 40, edge)
0xF100_0000 .. +0x10_0000 PCI ECAM, bus 0
Linux boots the arm64 way: the kernel is at a 2 MiB-aligned base, x0 holds the
device tree, the MMU is off, and everything runs at EL1h. Kernels can be raw Image,
gzip, zstd or EFI zboot (kernel.rs). They come from the host, or from /boot in
the 9p filesystem (Vm::boot_from_fs).
PCI slots are fixed: 1 = console, 2 = 9p (tag host9p), 3 = network. Each device
is a modern virtio-pci function with one memory BAR (common config, ISR, device
config, notifications) and a legacy INTx line. The device tree marks the bus
linux,pci-probe-only, so Linux keeps the BARs armless assigned.
Code map#
cpu.rs, a64.rs, advsimd.rs, crypto.rs, fp.rs, simd*.rs, mmu.rs, sysreg.rs |
the CPU: state, decode and execute, the MMU and system registers |
predecode.rs, cpu/interp.rs |
the interpreter: each code page decoded once to ops, and the loop that runs them |
jit/ |
hot code pages translated to wasm (gen.rs): run as wasm in the browser, compiled to host code with Cranelift natively (native.rs, the default); profile.rs carries what a machine compiled to the next |
cpu/hw.rs, cpu/hv.rs, cpu/kvm.rs |
the CPU in hardware (hypervisor feature): Hypervisor.framework, KVM |
gic.rs, rtc.rs, smp.rs |
shared machine state, multi-CPU GIC, PSCI, TLB shootdowns |
board.rs, dtb.rs, ram.rs |
the board's memory map and its device tree, guest RAM |
dev/ |
UART, PCI, virtio transport, console, 9p + filesystem, net + network stack, the web relay (dev/web/) |
vm.rs, state.rs |
Board (devices + host API), Vm (with CPUs), snapshots and their CPU/machine state |
runner.rs, native/, platform.rs, spill.rs |
native hosting: threads, blob directory, egress, TCP NAT, decompressed blobs on disk |
kernel.rs |
a distribution's arm64 kernel made a raw Image |
wasm.rs, heap.rs, js/armless.js |
the browser build |
../user/ |
user mode: a Linux program with no guest kernel (armless-user, armless-exec); its wasm.rs adds user mode's exports to armless.wasm |
map.md says where each feature and each system call lives, and which tests cover what.
Host interface#
The machine never calls the host synchronously for I/O. What the host has to do
is queued as Events, and the host answers through Board:
| event | answer |
|---|---|
Serial(bytes), Console(bytes) |
(display it) |
Blob(name) |
provide_blob(name, contents) or blob_failed(name) |
HttpRequest { id, method, url, headers, body } |
http_respond (status, headers), http_body…, http_end; or http_fail |
WsOpen { id, url, protocols } |
ws_accept(id, protocol) or http_fail; then ws_send/ws_close |
WsMessage, WsClose |
(deliver to the other end) |
TcpOpen { id, host, port } |
tcp_accept(id) or tcp_refuse(id), then tcp_send/tcp_close |
TcpData, TcpEof, TcpClosed |
(deliver to the other end); with flow control, tcp_consumed(id, n) once TcpData is sent on |
TcpConnected { id } |
after tcp_connect(port), the host's connection into the guest is up |
Input goes through console_input, console_resize and serial_input. Console
input is queued until the guest posts receive buffers, so nothing is dropped.
A 9p request that needs a blob that hasn't arrived waits (its descriptor chain is held) and is answered when the blobs it needs have all arrived. In the browser, fetched blobs are cached up to 256 MiB, oldest evicted first, except the chunks a waiting read covers. A native host decompresses each blob once into an unlinked temporary file (mode 0600, in the temporary directory) and reads it with pread, so the blobs take no heap and are never decompressed twice. When the temporary directory is tmpfs or ramfs, or its file system cannot be told, or the file cannot be made or fills up, the blobs stay in the cache above (the host prints one warning for a file that fills up).
The filesystem#
filesystem.json is v86's format 3: fsroot lists [name, size, mtime, mode, uid, gid, target], where target is a directory's children, a symlink's target, or a
regular file's blob. A file can instead be in chunks, [[blob, length], ...], whose
lengths add up to size:
["big.bin", 8388608, 0, 33188, 0, 0, [["0e2c9a7d41.bin.zst", 878612], ["...", 7509996]]]
A read fetches only the chunks it covers, and a write copies only the chunks it
touches into memory (truncating keeps a prefix of a chunk without fetching it).
Chunks are plain blobs to the host, so identical chunks are shared across files and
images. tests/fs/cdc.py cuts them where the contents say (a gear hash, 256 KiB to
4 MiB, about 1 MiB), so an insertion only changes the chunks around it.
The network#
The guest is 192.168.86.100/24, and the router 192.168.86.1 answers ARP, DHCP, DNS and ping. DNS gives every name an address of its own (10.x.y.z, kept in snapshots), so the machine always knows which name a connection is for.
The web relay (dev/web)#
Browsers and most sandboxes only let HTTP out: fetch() and WebSocket, or an HTTP(S)
proxy. So the guest's connections to port 80, and to 443 once Board::set_tls gave
the site's CA, end inside the machine:
- TLS 1.3 on 443 (
tls.rs) uses TLS_AES_128_GCM_SHA256 with X25519 or P-256 and an ECDSA P-256 certificate minted per SNI name, signed by the site's CA (cert.rs). The guest trusts that CA. There is no HelloRetryRequest. Every current OpenSSL, Node, Python, Go, Rust and Java client offers what it needs. - Resumption: every handshake ends with a session ticket (valid a day, for that SNI
name). A client that comes back with it skips the certificate and its signature, but
still makes a fresh X25519 or P-256 exchange (psk_dhe_ke only; no 0-RTT). Tickets are
sealed with a key made from the host's seed and forgotten at every
Board::seed, so copies of one snapshot can't use each other's tickets; a ticket the machine can't open, or one that is too old or for another name, just means a full handshake. A wrong binder fails it. - HTTP/1.1 inside (
mod.rs) keeps connections open: Content-Length or chunked bodies,Expect: 100-continue, and requests answered one at a time (a pipelined one waits in the buffer, and is a newHttpRequestonce the response before it ends). Responses are streamed with the host's Content-Length, or chunked when it gave none (fetch() never does); a host that sends less, or more, than it said closes the connection. HTTP/1.0 andconnection: closerequests get a response that ends with the connection, as does a guest that has finished sending, after its last request. Requests are parsed strictly, since the host passes them on (perhaps to a shared proxy): a bare CR or LF or another control character in the head, a folded line, a header name or method that is no token, or a Content-Length that isn't one plain number is a 400. A body may be at most 64 MiB (MAX_BODY, refused with 413 as soon as the head or a chunk size says more); a chunked one is read incrementally. What the guest sends while its request is answered, or its WebSocket opened, is held up to a body and a head more (pipelining); past that the connection closes. - WebSocket upgrades become
WsOpenand the frames become messages. Pings are answered in the machine. A frame or a message over 64 MiB (MAX_WS_MESSAGE), from the guest or from a server the native host talks to, closes the WebSocket with 1009, as soon as the frame's length is in.
The host only sees HttpRequest/WsOpen, with full https://, wss://… URLs:
- Browser:
fetch()andWebSocket, upgraded to https:// and wss:// on https pages (CORS applies). Requests never carry the visitor's cookies, authentication or referrer, and private destinations are refused unless the page allows them (allow,allow_private); under Node, Bun and Deno names are checked by what they resolve to, and redirects go back to the guest instead of being followed. - Native:
native::Egress, a small HTTP/1.1 + WebSocket client on rustls. It goes direct or through the environment's proxy (CONNECT for TLS, absolute-form for plain HTTP,NO_PROXYhonoured) and checks TLS against the system roots, so TLS-intercepting sandbox proxies work. Private addresses are refused unless allowed: going direct, every address a name resolves to is checked and only those are dialled (so DNS rebinding can't slip past); through a proxy, which resolves names itself, destinations that are private as written (IP literals in any forminet_atonreads,localhostand*.localhost) are refused before the proxy sees them, and names are the proxy's own policy.
Randomness for TLS comes from the host (Board::seed): the browser uses
crypto.getRandomValues, natively it comes from /dev/urandom. It is seeded again
after every restore.
The same seed gives the guest a new VM generation ID (microsoft,vmgenid in the device
tree) and raises its interrupt: Linux reseeds its random generator at once
(add_vmfork_randomness). Without it, every restore of a snapshot would draw the same
"random" numbers (UUIDs, keys, nonces) as the machine that was saved.
Other TCP#
Other ports reach the host as TcpOpen streams:
- Native: a real connection, direct or tunnelled with CONNECT through the proxy.
- Browser: refused unless the page takes it.
The guest has at most 1024 TCP connections through the stack at once (MAX_CONNS, relayed
web connections included); past that a SYN is reset, as by a full listen queue, so a guest
can't make the host open connections (natively, threads) without end.
Flow control. A host that turns it on (Board::set_tcp_flow_control; NativeHost and armless.js
do) says what it has done with each connection's data (tcp_consumed(id, n): sent on, or
dropped), and the guest is offered a window of what is left of 64 KiB unconsumed: a guest writing
to a far end that reads slowly, or not at all, waits in its own socket buffer rather than piling
its data up in the host. Data past the window is dropped unacknowledged, as from a guest whose
stack ignored it, and the guest gets a window update once there is room for a segment. NativeHost
says a write's bytes are consumed once the socket took them (a tunnel's, once its WebSocket
frame is written); armless.js once a data listener had them, or for a tunnel once the
WebSocket's bufferedAmount drained. A host that never says would stall its connections, so the
stack's window stays open unless it is turned on. The web relay's connections are consumed in the
machine (and bounded there).
Either host can carry chosen destinations over a WebSocket instead: a tunnel maps a
name and port (or any name, on a port) to a ws:// or wss:// URL, and each connection to
it becomes one WebSocket whose binary messages are the stream's bytes (what websockify,
websocat and Neon's wsproxy speak). It is HostOptions::tunnels natively (Tunnel::parse
reads [HOST:]PORT=URL) and network.tunnels in the browser, where it is the only way
out for raw TCP, and where the URL can be made per connection by a function (a fresh
access token for a gateway that holds the real credentials, so none is in the page). The guest's SYN waits until the WebSocket is open, so a tunnel that
can't be reached is refused like a closed port. A WebSocket can't half-close: the guest
finishing its side closes it.
With the network off, everything is refused.
TCP towards the guest respects its window. The link never drops frames: they wait
for receive buffers. A retransmit timer (Board::tick) covers what the guest's own
stack drops. Connections are not part of snapshots: after a restore, the guest gets
RST for any it still has.
Threads#
Each CPU runs in its own host thread or Web Worker. Devices run on whichever thread
touches them, under one lock: a vCPU doing MMIO, or the host delivering input. Idle
CPUs sleep on a wake counter (Atomics.wait in workers) until their timer expires
or something kicks them. Device DMA into a page holding translated code invalidates
that code on every CPU (Machine::code_pages).
With several CPUs, a store-exclusive succeeds if memory still holds what its load-exclusive read (a host compare-and-swap, as QEMU does), and the LSE atomics (CAS, SWP, LDADD...) are host compare-and-swap loops on guest RAM, so both are atomic against every other CPU. 128-bit pairs (STXP, CASP of X registers) are updated under one machine-wide lock. Unaligned atomics take an alignment fault; the JIT leaves atomics and the crypto instructions to the interpreter.
For a snapshot every CPU is paused between instruction slices. A paused CPU counts as idle, so a running CPU waiting for TLB acknowledgements does not wait for it.
Waiting for another CPU#
A CPU waits for another in three places: a DSB after a broadcast TLBI (until every running CPU
applied it), the guest's WFE (its spinlocks, and software polling), and a SpinLock of
armless's own (the GIC, the RTC, the TLB and code rings). How it waits is Machine::policy:
- Adaptive (the default): a TLB wait spins for
SPIN_NS(50 µs), then gives up its host thread now and then; a guest's WFE wait (followed over the many WFEs of its loop, and ended when the CPU goes idle) does the same once it has lasted that long; aSpinLockgives up its thread after 128 spins, on a thread that may block (a vCPU worker, not a page's own thread). A TLB wait ofSLOW_WAIT_NS(2 ms) or more means the CPU waited for had no host core, and after three the machine counts as oversubscribed whatever the host reported, so waits give up their thread almost at once. (Only TLB waits teach this: a guest may wait in WFE for seconds on purpose.) - Spin-count (armless 0.12): spin a set number of times (100,000 for a TLB wait, 4,000 for a WFE), fewer when the host reports no more cores than CPUs.
What the waits cost is in the memory report's smp (below), with how many instructions each
CPU ran.
Snapshots (format 3)#
"ARMLESS\x03" u64 RAM size u32 CPUs u64 meta length
meta: CPUs, GIC, timers, RTC (state.rs), then the devices (UART, PCI functions,
virtio queues, console, the filesystem tree with files the guest wrote
(of a chunked file, only the chunks it wrote),
open 9p fids, waiting requests, network names), then tagged extensions
(u32 tag + data; "P9OW": who each 9p fid belongs to, "VGID": the VM generation ID)
RAM: (u32 page, 4096 bytes) for every non-zero page, then u32 0xffffffff
StateLoader streams this, zstd-compressed or not, straight into guest memory.
Blobs are not saved; the guest's unmodified files are fetched again after a restore.
Chunked files are a new content kind in the tree, so older armless can't load a snapshot
that has one; snapshots without them are unchanged.
Extensions go after the devices, so snapshots without them still load (9p fids then
belong to root, as they used to).
A machine can be saved as one of its first n CPUs (save_state_cpus) once the guest has taken
the others offline (echo 0 > /sys/devices/system/cpu/cpuN/online, which ends in PSCI CPU_OFF):
their state is left out and shared interrupts aimed only at them go to CPU 0. A machine with n
CPUs restores it; the guest sees the rest as offline CPUs, and CPU_ON for them fails. snowglobe
boots and warms a site on more CPUs than it ships with this way.
Pages a Linux guest has freed still hold what they held, so a snapshot after a program allocated
and gave back a lot of memory carries all of it. With free page reporting
(DeviceConfig::free_page_reporting, armless.js's free_page_reporting, armless-run's
--free-page-reporting) a virtio-balloon sits in PCI slot 4 offering only
VIRTIO_BALLOON_F_REPORTING: the guest's kernel (CONFIG_VIRTIO_BALLOON, CONFIG_PAGE_REPORTING; the
test kernel's armless_defconfig has both, as do Ubuntu's generic kernels) reports free blocks of
pageblock order (2 MiB) a couple of seconds after they free up, and the device zeroes them
(Io::discard: the host's pages go back where they can, and code compiled or decoded from them is
dropped on every CPU). tests/linux/free-pages.sh (in CI) boots the test kernel with an init that
writes and frees 200 MiB in a 512 MiB machine: on 2026-11-17 its snapshot was 227.4 MB without
free page reporting and 59.5 MB with it. It is off by default because the devices are
part of what a snapshot must be restored onto. A range is zeroed only where it is all RAM, and a
guest that reports a page it still uses only loses its own data.
SMP in the browser#
armless-smp.wasm is built with atomics and imports a shared memory. The page's
instance creates the machine and serves host calls. Each vCPU worker instantiates
the same module on that memory, with its own stack, TLS and JIT function table, and
calls armless_run in a loop. The workers raise host_notify when events are
waiting. A shared control block parks every worker for save_state.
A worker compiles a JIT block in place: new WebAssembly.Module and new WebAssembly.Instance,
which a worker may do at any size (a page's own thread may only for small modules). The block
is marked ready as soon as armless_run returns, with no turn of the event loop. A compilation
the browser refuses that way goes through WebAssembly.instantiate, which finishes only when
the worker's event loop runs; a worker waiting in Atomics.wait or running the guest doesn't
run it. So a worker with such compilations under way lets its event loop run (a posted message
to itself) every 4 ms, and one with none doesn't.
Why it matters: a turn of a worker's event loop costs well under a millisecond in Chrome but around 25 ms in Safari on an iPhone. Workers that yielded every 4 ms whatever they did spent about 80% of their time there, and a 2-CPU Supabase couldn't create a table within 90 s. With yields only while compilations were under way, the same machine ran each step of its setup in 1 to 2 s, and a 3-million-row query in 34 s (43 s with the fixed spin count); the 1-CPU site, which has no workers, ran it in 15 s. Compiling in place takes the remaining yields away: on a desktop, a boot and the same steps compiled about 3,000 blocks with no yield at all.
Each worker counts where its time went in a shared buffer: running the guest, waiting idle,
yielding, slices, idle waits, events posted to the page, yields and compilations. The page
counts how late the workers' event messages reach it (a device request waits that long).
memory_report().smp and smp_report() give them with the waits above.
User mode (experimental)#
user/ (the armless-user crate, and armless-exec on the command line) runs a Linux program
with no guest kernel. The machine is the same; what is missing is EL1. The CPU runs the program
at EL0 with no vector table and Cpu::stop_on_exceptions set, so every exception it takes stops
the CPU (StopReason::UnhandledException, cpu.rs) with ESR, ELR and FAR, whether the MMU is on
or off. That is true in the interpreter and in both JITs. The host reads the syscall from x8 and
x0–x5, answers it in x0, and resumes at ELR.
- Programs: static or dynamically linked. A dynamic one's loader (PT_INTERP, musl's
ld-musl-aarch64.so.1say) comes from the machine's filesystem, as execve has it, and maps the libraries itself;Process::execfinds the program by PATH. - Files: the machine's filesystem, the one the 9p device serves a Linux guest
(
Board::with_fs): open, read and write (by offset too), stat and statx, getdents64, readlink, getcwd and chdir, mkdir, unlink, rename, symlink, link, truncate; paths resolve through symlinks and... File contents still arrive as blobs, when first read: a read that needs one parks its thread, the machine asks the host for it (Board::want_blobs), and the call runs again once it is there, as a 9p request does. Descriptors are Linux's: open file descriptions with their offset, shared by dup./dev/null,zero,urandomandtty, and a few/procfiles, are made here. File mappings (mmapof a file) are private copies, filled lazily (below). - The terminal: stdin, stdout and stderr are a terminal with termios (TCGETS, TCSETS), a
window size (
Process::resize, TIOCGWINSZ) and a line discipline, as n_tty: in canonical mode a line is edited and echoed before it is read; in raw mode bytes pass through; with ISIG, Ctrl-C, Ctrl-\ and Ctrl-Z are SIGINT, SIGQUIT and SIGTSTP, flushing the input and the output not taken yet (unless NOFLSH) and echoing^C; a new size is SIGWINCH; output newlines start a line (ONLCR). - Without a terminal:
ExecOptions { std: StdStreams::Plain, .. }(Process::exec_with;armless-exec --plain-std;armless_user_plain_std;plain_stdin the JS options) starts the program as a container with no tty has it: stdin is/dev/null(end of file,isattyfalse), stdout and stderr are character devices that are no terminal (no TCGETS, no ONLCR) and write to the same output (take_output, console events), and/dev/stdin,/dev/stdoutand/dev/stderropen the same; with no controlling terminal,/dev/ttyis ENXIO.ExecOptions::hostname(--hostname,hostname) is what uname and/proc/sys/kernel/hostnamesay ("armless" by default). Both are in the saved state. - Orphans: a process whose parent ended is a child of the first process, which can wait for
it. A container's program is pid 1 and does not:
Process::reap_orphans(armless-exec --reap-orphans,armless_user_reap_orphans,reap_orphansin the JS options; saved) makes the first process reap them as init does. They get no SIGCHLD for it,wait4(-1)andwaitidnever report them (ECHILD once it has no children of its own), and they leave as they end, so nothing piles up as zombies. Its own children are as before. Without it, as Linux reparents, an orphan ends with SIGCHLD, and one that had ended already raises it in the first process then (or goes, if that one ignores SIGCHLD). - Pipes, eventfd, epoll (edge-triggered and one-shot too), poll and select.
- File locks (user/src/locks.rs), as Linux keeps them: record locks (
fcntlF_GETLK, F_SETLK, F_SETLKW) are a process's, on byte ranges, and go when it closes any descriptor of the file or ends; OFD locks are a description's and conflict with them;flockis a description's whole-file lock. SQLite's locks hold, and the Supabase CLI tells its stack's owner is running by one. - Sockets (user/src/net.rs):
socket,socketpair,bind,listen,acceptandaccept4,connect,sendtoandrecvfrom,sendmsgandrecvmsg(and themmsgpair),shutdown,getsockname,getpeername,getsockoptandsetsockopt; a socket is a descriptor like a pipe's, and read, write, readv, writev, fcntl, poll, select and epoll take it. Blocking and non-blocking (SOCK_NONBLOCK,EAGAIN,EINPROGRESS) as for pipes: a call that can't finish parks its thread and runs again when something changed.FIOASYNC(O_ASYNC) andF_SETOWNare accepted, and SIGIO is never sent.- In memory: AF_UNIX stream and datagram sockets (paths, which are entries of the
filesystem, and abstract names, and
socketpair), and AF_INET and AF_INET6 TCP and UDP to the loopback (127/8,::1, the machine's own address, an unspecified one), through a table of listening ports the process keeps. Streams buffer 64 KiB for the reader, datagram queues hold 128; a stream's end that closes gives the other EOF, a write to it EPIPE, a connect to a port nobody listens on ECONNREFUSED. Ports an unbound socket gets start at 49152. - The host: on a machine with a network, a connection to any other address is a host
stream (
Event::TcpOpen, with the name the DNS answer gave for a 10.x address, else the dotted address), accepted, refused or carried over a tunnel as for a Linux guest;connectwaits for the answer. A guest listener is reachable from the host (Board::tcp_connect(port), whicharmless-exec --publish [ADDR:]HOSTPORT:PORTand any host's forward use): the machine keeps the listening ports and the connections made to them (core,dev/netstack/local.rs), not the guest netstack's SYNs. A UDP datagram to port 53 of a non-local address is a DNS query the network stack answers (Board::local_dns), so resolv.conf's nameserver works; other UDP to the outside and all IPv6 to the outside fail with ENETUNREACH. - Processes: the sockets are the machine's, like the pipes: a descriptor refers to one
through its description, so a fork's child has its parent's sockets (a connected one, a
listener it can
accepton too),execvekeeps them unless they are close-on-exec, and a call that waits on one (of any process) runs again when it changes. - Snapshots: the sockets' state is in the machine's, streams and queues with their bytes, whatever process holds them. A connection to the host can't survive a restore: it reads as closed, and one that was connecting as reset.
- Ancillary data goes between unix sockets, streams and datagrams.
SCM_RIGHTSpasses descriptors (Erlang hands its port programs' pipes toerl_child_setupso): a datagram carries them, a stream the bytes of the send that had them, each holding a reference until received;recvmsginstalls them (MSG_CMSG_CLOEXEC, MSG_CTRUNC when its room is short), a peek gives copies, and a read orrecvfrom, which takes no control data, closes them. A stream read that reaches a send carrying descriptors takes them and ends with that send, as Linux's. WithSO_PASSCREDa socket receivesSCM_CREDENTIALS, who sent what it reads (pid, uid, gid, or what the sender gave, as root may; a pid that isn't a process's is ESRCH). An unbound unix client is accepted as its family alone (Postgres's "local" lines match so). - Interfaces (user/src/netlink.rs): a netlink route socket answers the RTM_GETLINK,
RTM_GETADDR and RTM_GETROUTE dumps (
ip addr,ip route, getifaddrs, Node'sos.networkInterfaces(), Go'snet.Interfaces(), Erlang'sinet:getifaddrs()):lo(127.0.0.1/8, ::1/128) and, with a network,eth0at 192.168.86.100/24 with the default route via 192.168.86.1; a change is EPERM. Any socket answers the SIOCGIF* ioctls from the same table (ifconfig), and/proc/net/devlists the interfaces. - Not supported: descriptors to the host (
EOPNOTSUPP), other ancillary data of a unix socket (EINVAL),SOCK_SEQPACKET, raw and packet sockets but netlink's route family,socketpairfor anything but AF_UNIX, a separateRDHUP, backpressure from the guest to the host (a host stream's data is taken as it comes).
- In memory: AF_UNIX stream and datagram sockets (paths, which are entries of the
filesystem, and abstract names, and
- Memory (user/src/mm.rs): virtual. Each process has page tables of its own (4 KiB pages,
39 bits: 512 GiB of address space, an ASID per process so a switch flushes nothing) over one
pool of refcounted frames, the machine's RAM; the tables are in frames too, which no process
maps. A process's areas say what each range is: its protection, whether it is shared, and the
file its pages come from if any. The ELF's segments go where they ask (a PIE, and the loader,
at a base of their own),
brkafter them, the stack at the top of the space andmmapbelow it, from a free list;mremapmoves or grows by moving page table entries, not pages. Protections hold: a store to a read-only page, a jump to one that is not executable and any access to PROT_NONE are faults, and a system call cannot write where the program cannot (EFAULT). The address space has nothing to do with the RAM: a page has a frame when something first touches it (a translation fault stops the CPU, the host takes a frame, zeroed or filled from its file, maps it, and resumes at the instruction). So a reservation costs nothing: a PROT_NONE or MAP_NORESERVE mapping of 8 GiB on a machine of 128 MiB (what V8's heap cage and WebAssembly memories ask for, and Go's 600 MB) takes no frame,mprotectand touches take frames where the program uses them, andmunmapandmadvise(MADV_DONTNEED)give them back (munmapthe page tables with nothing left in them too; a frame's memory goes back to the host withRam::discard). There is no overcommit check: a program that touches more than the RAM has is stopped when the pool runs dry, not when it maps. - Lazy file pages: the executable's segments, its loader's, and every
mmapof a file are areas with a file behind them, not read: a page gets its contents when something first touches it. So a snapshot holds the pages a program used, not every byte of every library it mapped, and only their blobs are fetched. The fault the touch is stops the CPU; the host fills the page from its file (or the thread waits for its blobs, as a read does, and touches it again), maps it and resumes. A system call that reads or writes a lazy page (awritefrom a mapped page) fills it first. A MAP_PRIVATE mapping is a copy: a write gets the page a frame of its own and the file keeps its contents. A MAP_SHARED one is the file's: every mapping of a page of it, in any process, has the one frame (Process::file_pages, Postgres's dynamic shared memory), and the page is written back to its file once no mapping holds it. The areas' bookkeeping is in the process state. POSIX shared memory (shm_open) is files in/dev/shm, a directory of the machine's filesystem a program's start makes if the image has none (Supabase's Postgres keeps its dynamic shared memory there), andfallocatesizes a file (mode 0, and KEEP_SIZE).ARMLESS_USER_EAGER=1reads every mapped page at once, for comparisons. - Threads: every guest thread is a saved register context, and they take turns on one
CPU in slices of instructions; a syscall answered on the way counts against the slice.
clone(threads, CLONE_THREAD), futexes (wait and wake, with timeouts and bitsets; requeue and wake-op, which musl's condition variables use),set_tid_addressandCLONE_CHILD_CLEARTID. A futex is its process's address, unless it is in a shared area and the call has no FUTEX_PRIVATE_FLAG: then it is its frame's, and waits and wakes meet across processes (Postgres's semaphores are in its shared memory). - Scheduler calls (user/src/sched.rs):
sched_setscheduler,sched_getscheduler,sched_setparam,sched_getparam,sched_setattr,sched_getattr,sched_get_priority_maxand_min,sched_rr_get_interval,setpriorityandgetpriorityanswer as Linux does for a process that may do them. Policy, priority and nice are kept per thread (PRIO_PROCESS acts on a tid, PRIO_PGRP and PRIO_USER on every matching thread), inherited byclone(SCHED_RESET_ON_FORK resets the child to SCHED_OTHER and a nonnegative nice) and kept by exec, and they are in the process state. SCHED_OTHER, BATCH and IDLE take priority 0, FIFO and RR 1 to 99; they are only kept and reported, the CPU is still shared by round robin. SCHED_DEADLINE is refused (EINVAL fromsched_setscheduler, EPERM fromsched_setattr).sched_rr_get_intervalsays 100 ms for SCHED_RR and 0 for the others.getpriorityreturns20 - nice, the raw call's convention. - Processes (user/src/proc.rs, exec.rs): a process is an address space, a descriptor table,
a working directory, signal actions and its threads; the descriptions the descriptors refer
to, the pipes, the terminal and the clock are the machine's, shared by all, so a pipe joins a
parent and its child.
clonemakes one withfork's flags, asvfork(the parent waits until the child execs or ends) or with CLONE_VM, andclone3too; the address space is copied on write (the tables are copied, not the pages: both sides map the frames read-only, and the first write by either takes a frame of its own, or makes the page writable again when the other side is gone).execveandexecveat(an fd, AT_EMPTY_PATH) run another program in the same process: a new address space, the descriptors that are not close-on-exec, handled signals back to the default, the other threads gone, and a#!script runs its interpreter (so does the first program,Process::exec: an image's command is often a script); a failure (ENOENT, EACCES, ENOEXEC, E2BIG, ELOOP) leaves the program as it was, and one whose file hasn't arrived waits for it and runs again.wait4andwaitid(WNOHANG, a pid, a group or any child; a zombie keeps its status),killof a pid, and SIGCHLD when a child ends (a parent that ignores it leaves no zombies). A process'sexit_groupends only it; its children go to the first process, and the machine ends when that one does. Not yet: the descriptor table, signal actions and working directory shared by CLONE_FILES, CLONE_SIGHAND and CLONE_FS (copied instead), futexes in the private memory of CLONE_VM processes (they wake only within a process), the siginfo of SIGCHLD (its pid and status), job control (stop and continue), namespaces, and an anonymous MAP_SHARED area that was PROT_NONE at a fork shares nothing (the others have their untouched pages made at the fork). - Users (user/src/creds.rs): a process has real, effective and saved user and group ids,
root to start with.
setuid,setgid,setreuid,setregid,setresuidandsetresgidchange them as Linux does (root to anything, anyone else among its own),getuidand the rest say them, a child has its parent's andexecvekeeps them, and what a process makes (files, directories, links, sockets' names) is owned by its effective ids.chownandfchownatset a file's owner (root to anyone, its owner to one of its groups). A process has supplementary groups too:setgroups(root's) keeps a listgetgroupsgives, a child inherits and exec keeps. Sosuand Postgres'spg_ctl(which refuses to run as root) work. Permissions are not checked by them. - System V shared memory (user/src/shm.rs):
shmget,shmat,shmdt,shmctl(IPC_STAT, IPC_SET, IPC_RMID). A segment is the machine's: it has frames of its own, each attach maps them as a shared area, and a fork shares them; it goes once removed and detached. - signalfd:
signalfd4gives a descriptor whose reads take the pending signals of its mask, which the program blocks (asignalfd_siginfoeach, a timer's with its id and overrun), readable to poll and epoll. - Timers (user/src/timer.rs, timerfd.rs): the interval timers (
setitimer,getitimer: ITIMER_REAL raises SIGALRM; ITIMER_VIRTUAL and ITIMER_PROF, SIGVTALRM and SIGPROF, from the CPU time of the process, the host time its threads held the CPU) and the POSIX timers (timer_create,timer_settimewith TIMER_ABSTIME,timer_gettime,timer_getoverrun,timer_delete: SIGEV_SIGNAL, SIGEV_THREAD_ID, SIGEV_NONE, the SI_TIMER siginfo and overruns) on the wall, monotonic, boot and CPU-time clocks (CLOCK_PROCESS_CPUTIME_ID,CLOCK_THREAD_CPUTIME_ID, forclock_gettimetoo). A child has no timers; exec keeps the interval timers and deletes the POSIX ones. A timerfd (timerfd_create,timerfd_settime,timerfd_gettime; GHC's runtime, Supabase's PostgREST, makes one) counts its expirations: a read gives them, poll and epoll see one that fired. An idle machine's host comes back when the next timer is due. - Process descriptors:
pidfd_openandCLONE_PIDFD(clone and clone3) give a descriptor of a process, readable once it has ended and hung up once it was waited for;pidfd_send_signalsignals it,pidfd_getfdtakes a description of its table,waitid(P_PIDFD)waits for it (one with PIDFD_NONBLOCK never waits), and/proc/self/fdinfo/Nsays its pid (the Supabase CLI watches its stack's owner so). - AIO (user/src/aio.rs):
io_setup,io_destroy,io_submit,io_cancel,io_geteventsandio_pgetevents, done as they are submitted (reads, writes, their vectored forms and fsync), their completions signalled to an eventfd with IOCB_FLAG_RESFD (nginx'saio on).preadv2andpwritev2take RWF_NOAPPEND, RWF_APPEND and RWF_DONTCACHE (RWF_ATOMIC is EOPNOTSUPP), and a positioned write to a descriptor opened O_APPEND goes to its end, as Linux's. - CPU identification: a read of an ID register (
mrsof MIDR_EL1, MPIDR_EL1, the ID_AA64* registers), which EL0 can't make, is answered with the CPU's value as Linux answers it for a program (HWCAP_CPUID); Go'sx/sys/cpureads ID_AA64ISAR0_EL1 as it starts. - /proc/self:
exeis the file that runs with its links followed (musl's loader finds$ORIGINfrom it:/usr/bin/javalinks into the JDK);fd/Nsays what each descriptor is open on, as Linux does (a path,pipe:[N],socket:[N],anon_inode:[...]), for realpath in Zig, Bun and Deno;fdinfo/Nits position and flags. - Blocking: a call that can't complete (a read of the terminal or a pipe, a poll, a wait for blobs) parks its thread with its pc on the svc, and runs again when something changes, or returns at its deadline. So whatever a program waits on, a snapshot can be taken.
- Clocks:
clock_gettime,gettimeofday,nanosleep,clock_nanosleep. The monotonic clock goes on across a snapshot and its restores, by the time it was asleep when the host says (below). - Signals (signal.rs): actions per process, a mask, a pending set and an alternate stack
per thread, a pending set for the process (given to its first thread that doesn't block the
signal, the main thread first). A handler runs on the
rt_sigframethe kernel builds (siginfo, ucontext, sigcontext with an fpsimd_context record), on the alternate stack with SA_ONSTACK, andrt_sigreturntakes the thread back from it, mask and FP/SIMD registers too. With no SA_RESTORER (Go on arm64) the handler returns to a page of the process's with the vDSO's two instructions. A thread parked in a call is woken for a signal it handles: reads and writes of the terminal and pipes, and futex waits with no timeout, start again with SA_RESTART and return EINTR without; poll, select, epoll, sleeps (with the time left),rt_sigsuspendandrt_sigtimedwaitreturn EINTR. A wait for blobs is not interrupted (a disk wait): its signal comes once it ends. SIGCHLD, SIGCONT, SIGURG, SIGWINCH and the stop signals are ignored by default (no job control), the rest end the process with 128 + the signal.Process::killis a signal from the host. Go's SIGURG preemption is delivered. Not yet: queued real-time signals, the masks of ppoll, pselect6 and epoll_pwait, and signals for faults. - Snapshots: the machine's own snapshot holds RAM and the filesystem;
Process::saveholds the rest (the processes, their threads and waits, their memory maps and lazy mappings, descriptors and pipes, the terminal, the clock, signals), a few tens of KB; the frames and page tables are in the machine's RAM.Process::restoretakes it on a machine that loaded the snapshot. The state isARMLUSR7: 0.15'sARMLUSR6with each thread's scheduling, each process's groups and AIO contexts, the armed timers (CPU time counted so far, and the POSIX ones), the descriptors and senders of what waits at sockets, and timerfds;ARMLUSR6is 0.14'sARMLUSR5with each process's ids and interval timer, the System V segments, the shared file pages, and the descriptors on their way over sockets. Both older ones load (what they lack starts as a new process has it); a state from before 0.14 does not (addresses are virtual, there are processes, and the sockets are in the machine's state). A host that restores after a while says how long (Process::restore'sasleep,armless_user_asleep_msbeforearmless_user_restore, armless-exec's--restorethe age ofprocess.bin): the monotonic and boot-time clocks go on by that much, a wait or timer that came due in it is due at once, and a timerfd that repeats counts every period it missed, as if it had been read all along. A recurring signal timer (setitimer,timer_create) signals once, keeps its phase (its next expiry is on the period it had) and counts the other periods it missed as the overrun of a POSIX timer (but where the host's clock can't go back that far, as macOS's before it booted: then no overrun is counted, and the phase is kept only if it can go back a period). The sleep is at most about 68 years. The wall clock is the host's and the CPU-time clocks stand still. - The host:
host.rsis all user mode takes from the host besides the machine: a monotonic clock (Instant), the wall clock and randomness. Natively they are std's and the system's; in WebAssembly, whose std has no clock, they are armless's JavaScript imports (microtick,wall_clock_ms) and a SHA-256 generator keyed by the host's entropy (the lastarmless_seed, which armless.js gives after every restore).
Several CPUs#
On one CPU the threads of every process take turns (Process::run). On several (Smp, or
Process::run_on with all the machine's CPUs; armless-exec --cpus N) they run at the same
time: each CPU runs on a host thread of its own and runs guest code with no lock held. What a
thread asks of its kernel (a system call, a page it touches first, a fault) stops its CPU,
which takes the process's lock and answers it with the same code one CPU uses
(Process::answer); picking a CPU's next thread is under the lock too (Process::schedule),
and a CPU takes only a thread no other CPU has. Atomics need nothing more: a store-exclusive is
armless's, as with a kernel, and futexes are answered under the lock.
What one CPU does that another must see before it goes on:
- A change to an address space (unmapped, protected, copied on write, a fork): the
translations every CPU holds of what changed go before the call returns, by armless's
broadcast TLB invalidation (
Cpu::tlb_shootdown: the other CPUs drop them at their next instruction, and the call waits for those in guest code). - A process that ends or execs: its threads on other CPUs stop there first
(
Machine::preemptmakes a CPU'srunreturn at its next instruction), and their CPUs are out of guest code (Cpu::left_guest) before its memory is freed (take_off_cpus). - A signal for a thread another CPU runs: that CPU is stopped, and delivers it itself, its registers being on it.
A CPU with nothing to run waits on a condition variable that a CPU making a thread runnable
signals. A program is told of the CPUs there are (sched_getaffinity, /proc/cpuinfo; four at
least, as on one CPU) and getcpu says which it is on. A snapshot is taken with every CPU's
thread back in the process (Smp::pause), so it is the same whatever number of CPUs made it,
and restores on any.
Programs that do work in parallel go as much faster: four processes of Python 3.6 times, a DuckDB table scan 3.7 times, on four CPUs. A start that waits on itself does not: Supabase's stack is ready in 23 s rather than 25 s, its four CPUs running guest code a quarter of the time, the rest of it a chain of starts and the polls between them.
In the browser#
armless.wasm is the cdylib of armless-user, which depends on armless, so one module has both
export sets: armless's armless_* (the #[no_mangle] functions of a dependency are exported
from a cdylib as they are) and user mode's armless_user_* (user/src/wasm.rs). These run on
the machine armless::wasm keeps, with its vm(), out(), set_error() and pointer helpers.
armless-smp.wasm is still built from armless alone: in a page, user mode runs on one CPU. User mode adds
about 290 KB to armless.wasm (1.28 to 1.57 MB).
A host makes the machine as for Linux (armless_create, armless_fs_load, armless_seed)
and then, instead of a boot:
| export | |
|---|---|
armless_user_exec(argv, len, env, len, cwd, len) |
start a program (argv and env NUL-separated, each item ending with its NUL, an empty one kept as an empty argument): 0 started, 1 its files are being fetched (blob events: provide them, call again), -1 failed |
armless_user_plain_std(on) |
the next armless_user_exec has no terminal (StdStreams::Plain) when on is nonzero |
armless_user_hostname(len) |
room for the next exec's hostname (at most 64 bytes; 0 and armless_error otherwise): returns where to write its UTF-8 bytes, which armless_user_exec reads (empty: armless); the exec puts both options back to the defaults |
armless_user_restore(ptr, len) |
the process of a snapshot (Process::save's bytes), after armless_state_* restored the machine; for one restore, how long it was saved for is what armless_user_asleep_ms set |
armless_user_asleep_ms(ms) |
how long the next restored state was saved for, in ms (0, the default: none), which its clocks and deadlines go on by |
armless_user_run(budget_ms) |
0 its time ran out, 1 idle, 2 exited, 3 faulted (the reason in armless_error), -1 no process |
armless_user_wake_ms() |
after 1: the microtick() time of its next deadline, or -1 (only input or a blob wakes it) |
armless_user_waiting_on_blobs() |
1 when a thread waits on a blob rather than on input |
armless_user_waiting_for_input() |
1 when a thread waits in a call input could complete (read, poll, epoll) and none on a blob: a REPL at its prompt, not a program asleep |
armless_user_exit_code() |
the exit status, -1 while it runs |
armless_user_input(ptr, len), armless_user_close_input(), armless_user_resize(cols, rows) |
the terminal |
armless_user_reap_orphans(on) |
the first process reaps its orphans (after armless_user_exec, before it runs) |
armless_user_kill(sig) |
a signal for the program, as kill(2) sends it (SIGINT for a stop button) |
armless_user_save() |
the process's state, at armless_buf; 0 with the reason in armless_error when there is no process or a spawned one is still held (not released) |
armless_user_spawn(argv, len, env, len, cwd, len) |
start a process beside the program (Process::spawn: root, a session of its own, no parent; pipes for stdin, stdout and stderr whose host ends the module holds), as armless_user_exec takes its lists: its pid, -2 its files are being fetched (blob events: provide them, call again), -1 failed (armless_error). It runs while armless_user_run does |
armless_user_spawn_input(pid, ptr, len) |
bytes for its stdin: 1 accepted, 0 it has none any more |
armless_user_spawn_close_input(pid) |
the end of its stdin |
armless_user_spawn_poll(pid) |
takes what it did since the last poll: -1 for a pid that is unknown or released, else 0. Read it with armless_user_spawn_stdout() and armless_user_spawn_stderr() (each returns its length, the bytes at armless_buf) and armless_user_spawn_exit() (-1 while it runs, else 0 to 255, 128 + the signal); they answer for the last poll |
armless_user_spawn_pending() |
the spawned processes with output not taken or an end not given by a poll: their pids, 8 bytes each (little-endian i64) at armless_buf, the length in bytes. armless.js asks after each run and emits spawn-output |
armless_user_spawn_kill(pid, sig) |
a signal for its process group: it and what it started |
armless_user_spawn_release(pid) |
done with it: its pipes let go and, once it ended, the process taken away |
A spawned process that runs, sleeps or waits on files is work for the machine: armless_user_run
keeps running (or reports its deadline in armless_user_wake_ms), and armless_user_waiting_for_input
is 0 while one is at work. One only waiting for its stdin, which the host gives, is not. Snapshots
don't keep the host's ends of the pipes, so armless_user_save refuses while one is held;
armless.js wraps all of this in vm.spawn() (see the Readme).
What the program writes comes out of armless_events as console output, and the files it
reads that aren't there yet as blob events, so the event loop a host has for Linux serves it
too. A snapshot is the pair a native host makes: the machine's state, the bytes of
Vm::save_state (armless_save_*), and the process's, those of Process::save
(armless_user_save), restored machine first. A site snowglobe saved natively
(state.bin.zst and user.bin) restores in a page, and a pair saved in a page restores
natively.
Memory. Each process's resident pages are counted from its page tables (mm.rs: a counter
that rises when a page gets a frame and falls when it loses one, not saved: a restored process
counts what it has, and its peak is that): /proc/<pid>/status (VmSize, VmHWM, VmRSS,
RssAnon, RssFile, RssShmem, VmData, VmStk (the pages the stack uses), VmExe; VmPeak the largest size seen when areas were added or taken away),
/proc/<pid>/statm, /proc/<pid>/stat fields 23 and 24, and getrusage (ru_maxrss in kB
for RUSAGE_SELF, and for RUSAGE_CHILDREN the largest peak of a child that ended). A
copy-on-write page after a fork counts in each process, lazy file pages once they are filled, and
PROT_NONE and MAP_NORESERVE areas for the pages touched only. /proc/meminfo and sysinfo
free memory is RAM less the frames in use (pages and page tables). Against a guest kernel the anonymous
count is the same (Bun's REPL: RssAnon 4.3 MB here, 4.5 MB with a kernel) and the file-backed one
is smaller (12 MB, 33 MB): a kernel maps the pages around a fault that are in its page cache, this
maps the page that was touched.
Not yet: signals for faults (a fault ends the process, SIGSEGV is not delivered to a handler), more than one vCPU in a page, and the hypervisor backends (an SVC at EL0 traps to EL1 in hardware there, not to the host).
What runs, from snowglobe's examples' own filesystems: apk, curl (HTTPS through the host),
duckdb (C++), go (Bubble Tea, static), java (OpenJDK 21), python (the 3.14 REPL, 122
shared libraries, subprocesses), python-agent, rust (a static binary), typescript (tsx on
Node), iex (Erlang/OTP, with port programs), postgres (su, pg_ctl, the postmaster and its
processes, queries through psql: 100,000 rows in 1.2 s), elixir (a release talking to that
Postgres) and supabase (Postgres, PostgREST, Auth, Realtime and nginx from its own launcher:
built as a snowglobe site with --kernel none, ready 24 s after it starts, signup, row-level
security and the API on the machine's network address checked); a Python REPL saved with its
state and restored in 120 ms. user/corpus/survey.exs traces what each of snowglobe's
examples asks of a kernel, for what comes next.
Measured as a snowglobe site (a Go program with eight goroutines, macOS, the same
snowglobe-vm for each): the snapshot is 2.5 MB raw and 0.6 MB compressed, against 22.1 and
3.6 MB under a stripped Linux and 50.6 and 10.2 under Alpine's linux-virt; a machine's
footprint is 22 MB, against 57 and 136.
Lazy file pages, measured the same way on snowglobe's examples (compressed snapshot, before and
after): duckdb 15.6 MB to 4.9 MB (64 to 24 MB raw), python 4.3 to 3.1 MB, go 1.3 to 0.9
MB. Each answers its first command after a restore as fast as before when its host runs the
process again as soon as blobs arrive (pages first touched then are filled from the
filesystem).