nish/sync/shm

Memory that several processes map at once, and the atomics that order what they write to it.

import {
  ACQUIRE,
  ACQ_REL,
  RELAXED,
  RELEASE,
  SEQ_CST,
  SHM_ACCESS,
  SHM_EXISTS,
  SHM_INVALID,
  SHM_NOT_EQUAL,
  SHM_NOT_FOUND,
  SHM_TIMED_OUT,
  SHM_TOO_MANY,
  SHM_UNSUPPORTED,
  SHM_WOKEN,
  SharedRegion,
  ShmError,
  createRegion,
  fence,
  openRegion,
  unlinkRegion,
} from "nish/sync/shm";
import { createRegion, openRegion, unlinkRegion, ACQUIRE, RELEASE } from "nish/sync/shm"

// The process that owns it: a fresh, zeroed region of 4096 bytes.
const stale = unlinkRegion("jobs") // a region a crashed run left, if any
const region = createRegion("jobs", 4096).expect("create")
region.storeI64(0, toI64(42), RELEASE)

// Any other process of the same user: the same bytes.
const view = openRegion("jobs").expect("open")
const n: i64 = view.loadI64(0, ACQUIRE) // 42 once the store is seen

A region is a named POSIX shared-memory object (/dev/shm/<name> on Linux, shm_open("/<name>") on Darwin) mapped shared, so every process that opens the name sees the same bytes. A name is 1 to 30 of A-Z a-z 0-9 . _ -, not starting with ..

createRegion makes it with mode 0600 and refuses a name that exists (SHM_EXISTS), so a name another user planted is never adopted; openRegion refuses a region this user does not own, that anyone else may read or write, or (on Linux) that is not a regular file or has a second hard link (SHM_ACCESS). The name stays until unlinkRegion, even after every process has exited, which is how a process that starts later finds it; the mapping goes away with close or with the process.

Why named, and no anonymous region made before `fork`

A Nish program has no fork: spawn and nish/net/prefork start a program with exec, and every descriptor is close-on-exec, so neither an anonymous mapping nor a memfd reaches the child. A name does. The supervisor creates the region before preforkWith, every worker opens it by name, and the supervisor unlinks it after (examples/prefork-jobs.ts, docs/wp47-shared-memory.md).

Every access is checked

An offset is an i32 byte offset into the region. A 32-bit slot sits at a multiple of 4 and a 64-bit one at a multiple of 8, wholly inside the region; a byte copy fits the region and the buffer. Anything else panics, naming the offset and the region's size, exactly as an array index out of range does. An ordering an access cannot take (a load with RELEASE) panics too. A region is never read or written past its end.

Orderings are C11's

RELAXED, ACQUIRE, RELEASE, ACQ_REL and SEQ_CST; a load takes the first, second and last, a store the first, third and last, and a read-modify-write any of them. Each access is one LLVM atomic instruction on the speed and size profiles, where the runtime is inlined into the program; a compare-exchange that fails orders as C++ derives it (ACQ_REL fails ACQUIRE, RELEASE fails RELAXED). Bytes copied by read and write are plain copies, ordered by the atomics around them, and a copy that races a writer in another process can be torn but is never read past its window.

Waiting

waitI32 sleeps while a 32-bit slot holds a value and wake wakes the processes sleeping on it: Linux's futex, shared between processes. Darwin and wasi answer SHM_UNSUPPORTED (-95) from both, as setIdlePriority does on Darwin, and a program there polls instead.

A region belongs to the thread that mapped it

Under --threads the runtime's table of regions is thread-local, so a SharedRegion handed to another thread names nothing there and panics; that thread opens the name again. Native only: --emit-ts refuses a module that declares C functions, and creating a region on wasi answers SHM_UNSUPPORTED. Calling C is the ffi capability, so a program that imports this module is reported with ffi and --deny ffi refuses it.

Private helpers share the importing program's flat symbol namespace (docs/wp26-stdlib.md §3e), which is why each one carries the module's name.

Functions

createRegion

Creates the region name, bytes long and zeroed, mode 0600, and maps it. SHM_EXISTS when the name is taken: unlink a stale one first.

const createRegion: (
  name: string,
  bytes: i32,
) => Result<SharedRegion, ShmError>
namestringrequired

the region's name, 1 to 30 of A-Z a-z 0-9 . _ -, not starting with .; anything else is SHM_INVALID

bytesi32required

the region's length, from 1 to 1 GiB; outside that is SHM_INVALID

returnsResult<SharedRegion, ShmError>

a new SharedRegion mapping the fresh region, which this thread alone can use and which stays mapped until its close or the process ends; or the ShmError: SHM_EXISTS, SHM_INVALID, SHM_TOO_MANY when this thread has 63 regions open, code 95 on wasi, or the errno the system gave

fence

A fence of order, for the accesses around it: C11's atomic_thread_fence, so it orders the calling thread's accesses, to regions and to its own memory, exactly as that fence does; RELAXED is no fence at all.

const fence: (order: i32) => void
orderi32required

RELAXED, ACQUIRE, RELEASE, ACQ_REL or SEQ_CST; any other value panics

openRegion

Maps the region name that some process of this user created. Its size is the region's (on Darwin, rounded up to a page).

const openRegion: (
  name: string,
) => Result<SharedRegion, ShmError>
namestringrequired

the name the region was created under, with createRegion's rules (SHM_INVALID otherwise)

returnsResult<SharedRegion, ShmError>

a new SharedRegion on the same bytes every other mapping of the name sees, which this thread alone can use until its close; or the ShmError: SHM_NOT_FOUND, SHM_ACCESS for a region another user owns, that others may read or write, or (on Linux) that is not a regular file or has a second hard link, SHM_INVALID for a bad name or a region of size 0 or over 1 GiB, SHM_TOO_MANY, code 95 on wasi, or the errno the system gave

unlinkRegion

Removes the name, so no process can open it again; every mapping of it stays valid until it is closed. SHM_NOT_FOUND when there is none.

const unlinkRegion: (name: string) => Result<void, ShmError>
namestringrequired

the region's name, with createRegion's rules (SHM_INVALID otherwise)

returnsResult<void, ShmError>

Ok() once the name is gone, or the ShmError: SHM_NOT_FOUND, SHM_INVALID, code 95 on wasi, or the errno the system gave

Classes

SharedRegion

A mapped region. Every slot offset is in bytes from its start; an offset outside it, a misaligned slot or an ordering the access cannot take panics.

class SharedRegion
Member
new SharedRegion(handle: i32) Wraps a handle the runtime answered. A program gets a region from createRegion or openRegion instead; a handle that names no region open on this thread makes every access panic, and close answer code 9.
addI32(at: i32, delta: i32, order: i32): i32 Adds delta to the 32-bit slot at at, wrapping, and answers the value before, as one atomic read-modify-write.
addI64(at: i32, delta: i64, order: i32): i64 Adds delta to the 64-bit slot at at, wrapping, and answers the value before, as one atomic read-modify-write.
close(): Result<void, ShmError> Unmaps the region; the handle names nothing after. The name stays until unlinkRegion. Every window still lent from it is emptied first, so an index into one after this panics rather than reading unmapped memory (a window a push moved off the region is the program's own array, and is left alone); any other access through this object panics too.
compareExchangeI32(at: i32, expected: i32, desired: i32, order: i32): i32 Sets the 32-bit slot at at to desired if it holds expected, and answers what it held: the exchange happened when that is expected. It never fails spuriously (a strong compare-exchange).
compareExchangeI64(at: i32, expected: i64, desired: i64, order: i32): i64 Sets the 64-bit slot at at to desired if it holds expected, and answers what it held: the exchange happened when that is expected. It never fails spuriously (a strong compare-exchange).
exchangeI32(at: i32, value: i32, order: i32): i32 Sets the 32-bit slot at at to value and answers the value before, as one atomic read-modify-write.
exchangeI64(at: i32, value: i64, order: i32): i64 Sets the 64-bit slot at at to value and answers the value before, as one atomic read-modify-write.
giveBack(window: u8[]): i32 Empties window, which this region lent, and answers the length it had. After it the array has length 0 and may be passed to lend as reuse.
lend(at: i32, len: i32, reuse: u8[] | null = null): u8[] A u8[] of length len whose bytes are the region's [at, at + len), not a copy: an index or copyBytes into it reads and writes the region. A range outside the region panics. giveBack or close empties it (length 0), so an index after either panics. A push moves it into this process's own memory, after which it is no window and giveBack panics. Pass a window already given back as reuse to lend it again without a new header: each lend without one keeps a few bytes for the life of the process.
loadI32(at: i32, order: i32): i32 Loads the 32-bit slot at at, one atomic load.
loadI64(at: i32, order: i32): i64 Loads the 64-bit slot at at, one atomic load.
read(at: i32, dst: u8[], off: i32, len: i32): void Copies len bytes of the region at at into dst[off, off + len). A plain copy, not atomic: a writer in another process racing it can tear what it reads, and only the atomics around it order it.
size(): i32 The region's length in bytes.
storeI32(at: i32, value: i32, order: i32): void Sets the 32-bit slot at at to value, one atomic store.
storeI64(at: i32, value: i64, order: i32): void Sets the 64-bit slot at at to value, one atomic store.
waitI32(at: i32, expected: i32, timeoutMs: i32): i32 Sleeps while the 32-bit slot at at holds expected, at most timeoutMs (negative: no limit): SHM_WOKEN, SHM_NOT_EQUAL, SHM_TIMED_OUT, or SHM_UNSUPPORTED off Linux. It promises no ordering: load the slot again, with the ordering the caller needs, after it returns.
wake(at: i32, count: i32): i32 Wakes at most count of the processes sleeping on the 32-bit slot at at: how many it woke, or SHM_UNSUPPORTED off Linux.
write(at: i32, src: readonly u8[], off: i32, len: i32): void Copies src[off, off + len) into the region at at. A plain copy, not atomic: publish it to another process with a RELEASE (or SEQ_CST) store after it.

ShmError

Why a region could not be created, opened, unlinked or closed.

class ShmError
Member
new ShmError(code: i32) The error for one errno, with its text: strerror's, except that 95 (SHM_UNSUPPORTED negated) reads "not supported on this platform".
code: i32 The errno, positive: SHM_NOT_FOUND, SHM_EXISTS and the rest.
message: string What it means, such as File exists.

Constants

2

ACQUIRE: i32

A load that sees everything written before the RELEASE store it reads.

4

ACQ_REL: i32

A read-modify-write that is both.

0

RELAXED: i32

No ordering beyond the access itself being whole.

3

RELEASE: i32

A store after which everything written before it is seen by an ACQUIRE load of it.

5

SEQ_CST: i32

One order every SEQ_CST access in every process agrees on.

13

SHM_ACCESS: i32

ShmError.code: another user's region, one others may use, or not one region under one name.

17

SHM_EXISTS: i32

ShmError.code: a region of that name exists already.

22

SHM_INVALID: i32

ShmError.code: a name that is not 1 to 30 of A-Za-z0-9._-, or a size outside 1 byte to 1 GiB.

-11

SHM_NOT_EQUAL: i32

waitI32: the slot did not hold the value, so it did not sleep.

2

SHM_NOT_FOUND: i32

ShmError.code: no region of that name.

-110

SHM_TIMED_OUT: i32

waitI32: the time ran out.

24

SHM_TOO_MANY: i32

ShmError.code: this thread has 63 regions open.

-95

SHM_UNSUPPORTED: i32

waitI32 and wake on Darwin and wasi, and every region on wasi: not here.

0

SHM_WOKEN: i32

waitI32: woken, or for no reason the caller can see; load the slot again.