Builtins
No import is needed; builtins are resolved by name. The libc-backed ones may
also be imported from a nish: module, which is the same builtin under a name
that cannot be shadowed (see Builtin modules below).
Effects (none, read, write) are what the call does to the purity of the
enclosing function
(ARCHITECTURE.md).
Builtin modules (nish:)
The builtins backed by the C runtime can be imported instead of reached for as
globals. nish: is the only bare specifier the language accepts; but for
nish:secret and nish:region, which have source behind them
(Secrets, Regions), it
resolves to no file, and the import renames a builtin rather than introducing one —
the call checks through the same rule and emits the same IR as the global
spelling, which tests/run.js pins by compiling tests/cases/io_nish_import
and io_nish_import_global and comparing the two bodies.
| Module | For |
|---|---|
nish:fs |
files and directories |
nish:process |
processes, environment, clocks |
nish:io |
standard output and error, panic |
nish:secret |
secrets that are wiped |
nish:region |
region allocation |
nish:net |
sockets and the readiness loop |
nish:term |
terminal input |
nish:unsafe |
unchecked access, wrapping arithmetic |
nish:async |
tasks: spawn, await, wait, cancel |
nish:simd |
sixteen-byte vectors |
nish:fs exports readFileSync,
readFileSyncOrNull, readFileBytesSync, writeFileSync,
appendFileSync, writeFileBytesSync, renameSync, unlinkSync,
writeFileSyncMode, createFileExclusiveSync, mkdirSync,
mkdirSyncMode, isDirectorySync, readdirSync, realpathSync,
statMtimeSync.
nish:process exports exit (the global
process.exit), getenv, spawnSync, spawnSyncTo, spawnPiped,
spawnPipedWithin, spawn, spawnWithFds, kill, waitPid, availableParallelism,
totalMemory, setNice, setIdlePriority, setCpuAffinity,
monotonicNanos, wallClockNanos, signalFd, readSignal, argv,
platform, arch.
nish:io exports write, writeError,
writeBytes, panic.
nish:secret exports Secret, secret,
expose, exposeWith, wipe. It is the one module with source behind it
(Secrets).
nish:region exports Region, region,
regionWith, enter, enterWith, free. It has source behind it too
(Regions).
nish:net exports netAddress,
netLocalPort, tcpListen, tcpListenFlags, tcpAccept, netRead,
netWrite, netWrite2, netShutdown, netClose, netMss,
netSetOption and its option ids NET_TCP_NODELAY, NET_TCP_QUICKACK,
NET_SO_KEEPALIVE, NET_SO_SNDBUF, NET_SO_RCVBUF, tcpConnect,
connectResult, udpBind, udpSendTo, udpRecvFrom, pollCreate,
pollAdd, pollModify, pollRemove, pollWait, unixListen,
unixListenMode, unixConnect, unixAccept, unixSocketPair, sendFd,
recvFd, peerCredentials, netRingState and the constants
NET_UNIX_UNLINK_STALE, NET_UNIX_STREAM, NET_UNIX_SEQPACKET,
NET_LISTEN_RING. Each is a global too
(nish:net).
nish:term exports readStdin, waitStdin,
isTerminal, terminalSize, setRawMode, setTerminalRestore. Each is a
global too (nish:term).
nish:unsafe exports uncheckedGet,
uncheckedSet, wrappingAdd, wrappingSub, wrappingMul,
readCString, readStruct, cErrno. They are reached only through the import
(nish:unsafe).
nish:async exports spawn, awaitAll,
sleep, readable, writable, readableWithin, writableWithin,
taskId, cancel, cancelled, cancelAfter.
They are reached only through the import
(Async functions).
nish:simd exports the vector types u8x16,
u16x8, i32x4, u32x4, f32x4, f64x2 and their constructors,
splatU8x16 … splatF64x2, loadU8x16 … loadF64x2, store, extract,
replace; the masks mask8x16 … mask64x2; the lane operations add …
asF64x2, among them shl, shr, rotl and rotr. They are reached
only through the import
(nish:simd,
lane operations).
import { readFileSync } from "nish:fs";
import { argv } from "nish:process";
import { write } from "nish:io";
-
The forms are the ones every other import allows: a named import, with
as(tests/cases/io_nish_rename). Default, namespace, side-effect and type-only imports are refused under the same messages a relative specifier gets. -
An unknown module is
Unknown builtin module `nish:sqlite`(reject_nish_unknown_module); a name a module does not export isModule `nish:fs` has no export `statSync`(reject_nish_unknown_export). Both list what does exist. -
An imported builtin cannot be shadowed, and that is the reason to prefer
one. A global builtin is consulted only when no user function of that name is
in scope, so declaring one silently replaces it; declaring one that collides
with an import is
`panic` is already declared in this moduleinstead (reject_nish_import_clash). -
A name keeps its kind.
argv,platformandarchare values, and calling one is refused (reject_nish_property_call); the functions are not values: a builtin is not a capture-free top-level function of this program, so it cannot be a reference (reject_nish_builtin_value). -
exitterminates control flow exactly asprocess.exitdoes, so a non-voidfunction may end with it (io_nish_rename). -
Diagnostics name the canonical builtin whatever the call site calls it:
exit(1, 2)reports`process.exit` expects exactly 1 argument, which is the rule to look up. -
Everything else stays global —
Math.*, the width conversions,console.log/console.error,String.fromCharCode,Arena.*,OkandErr. They lower to an LLVM intrinsic or a single instruction and cost no runtime at all, so they read as language rather than as library.String.fromBytesis global too, besideString.fromCharCodeon the sameString, though both make their answer with onenish_str_new.Date.now()andcrypto.getRandomValuesare global too, though each is a runtime call, because JavaScript spells them as globals and a program that runs under Node unchanged has to find them where Node has them (The host).
console
The reference for each of these, with its parameters, its answer and its capability, is its doc comment in runtime/nish.d.ts; the rules are here.
| Function | Takes | Gives | Effect |
|---|---|---|---|
console.log |
string | number | i64 | f64 | boolean |
void |
write |
console.error |
as console.log |
void |
write |
write |
string |
void |
write |
writeError |
string |
void |
write |
writeBytes |
readonly u8[], i32, i32 |
i32 |
write |
panic |
string |
void |
write |
console.log(x: string | number | i64 | f64 | boolean): void writes x
and a newline to stdout. It is allowed in statement position only, and takes
exactly one argument.
- Lowering: one
write(2). -
Tests:
str_console_log,i64_basic;reject_console_log_noargs,reject_console_log_two_args,reject_console_log_as_value; arrays and objects are rejected (`console.log` accepts string, number, or boolean, got i32[](CLI only)).
console.error(x): void is the same, on stderr: the stream a
compiler's diagnostics belong on (WP14 B2). It has identical rules and
identical messages under its own name.
- Tests:
io_streams;reject_console_error_type.
write(s: string): void writes s to stdout with no trailing
newline and no conversion (a string only). It is allowed in statement
position.
- Tests:
io_streams.
writeError(s: string): void is the same, on stderr.
- Tests:
io_streams.
writeBytes(bytes: readonly u8[], off: i32, len: i32): i32 writes the
bytes of bytes[off..off+len) to stdout as they are. It answers len, or a
negative errno if the stream refuses them, never a panic. A pipe whose
reader has gone raises SIGPIPE natively, where Node answers -32 EPIPE. A
range outside bytes panics with
slice out of range: [off, off+len) of length n and exits 1 (a slice
site), under Node too. It needs no capability, as for write. A wasm32
build has no stdout to reach and refuses it.
-
Tests:
io_write_bytes,io_write_bytes_bounds;reject_write_bytes_arity,reject_write_bytes_type(NL2268),reject_write_bytes_wasm.
panic(message: string): void writes message and a newline to stderr,
then exits 1: the same ending an out-of-range index has. It is allowed in
statement position, and it terminates control flow like process.exit,
so a non-void function may end with it (WP14 D1, WP16).
- Tests:
io_streams;reject_panic_value.
Numbers print as JavaScript's String(x): exact decimal for i32/i64,
shortest round-trip digits for f64 (0.1, 1e+21, 1e-7, NaN,
Infinity, -0 as 0) (tests/runtime-test.c, tests/cases/str_f64_mode,
math_intrinsics). Booleans print true/false.
Reference: nish:io.
Math
The reference for each member of Math, with its parameters, its answer and its capability, is its doc comment in runtime/nish.d.ts; the rules are here.
| Function | Takes | Gives | Effect |
|---|---|---|---|
Math.sqrt/floor/ceil/trunc/sin/cos/exp/log |
f64 |
f64 |
none |
Math.tan |
f64 |
f64 |
none |
Math.atan2, Math.hypot |
f64, f64 |
f64 |
none |
Math.pow |
f64, f64 |
f64 |
none |
Math.round |
f64 |
f64 |
none |
Math.abs |
any numeric T |
T |
none |
Math.min, Math.max |
T, T |
T |
none |
Math.clz32 |
i32 | u32 |
integer<0, 32> |
none |
Math.sign |
any numeric T |
T |
none |
Math.imul |
two of i32 | u32 |
i32 |
none |
Math.random |
nothing | f64 |
write |
Math.PI, Math.E |
— | f64 |
none |
Math.sqrt/floor/ceil/trunc/sin/cos/exp/log(x: f64): f64 take and
answer an f64. floor, ceil and trunc have two paths on x86-64 (see
Lowering), and the answers are the same bits on either path.
-
Lowering: LLVM intrinsics (
llvm.sqrt.f64, ...), exceptfloor,ceilandtruncin an x86-64 module that is not a wasm build: there they call the runtime'snish_floor,nish_ceilandnish_trunc, which run SSE4.1'sroundsdon a CPU that has it and libm's routine on one that does not, chosen once beforemain(NISH_SIMD=scalarpins libm); with no--targetthat is an x86-64 host. On a page module (a wasm build with no WASI, or an--emit-dtscompile, whose IR--profile wasmorwasm-hostlinks with no libm),sin,cos,expandlogare calls tostd/math.ts'ssinF64,cosF64,expF64andlogF64instead, fdlibm's, copied in liketan, within one ulp of V8;--profile wasikeeps the intrinsics. Under--deterministicthey are those calls on every target,--emit-tsincluded. -
Tests:
math_intrinsics,math_rounding,wasm_math_exp,wasm_math_std;reject_wasm_math_f32(`Math.sin` requires an f64 argument, got f32);reject_math_i32(requires an f64 argument, got i32 (use --number-mode f64 or toF64(x))).
Math.tan(x: f64): f64 is exact near every multiple of π/2 and within
one ulp.
-
Lowering: a call to
std/math.ts'stanF64, fdlibm's, copied into the module asinternal @nish.tanF64, so no target calls libm. - Tests:
math_tan,math_std_edges.
Math.atan2(y: f64, x: f64): f64, Math.hypot(x: f64, y: f64): f64
follow ECMAScript's special cases (atan2(0, -0) is π, atan2(-0, -0) is
−π, hypot(Infinity, NaN) is Infinity). Math.hypot takes exactly two
arguments, not JavaScript's any number.
-
Lowering: calls to
std/math.ts'satan2F64andhypotF64, copied in liketan. -
Tests:
math_atan2,math_hypot;reject_math_atan2_f32(`Math.atan2` requires an f64 argument, got f32 (use --number-mode f64 or toF64(x))),reject_math_hypot_arity(`Math.hypot` expects exactly 2 arguments, got 3).
Math.pow(x: f64, y: f64): f64 gives ECMAScript's answer where it and
C99 differ: pow(x, NaN) and pow(±1, ±Infinity) are NaN (C gives 1);
pow(x, ±0) is 1, pow(NaN, 0) is 1, and every other special case
agrees with C.
-
Lowering:
llvm.pow.f64plus aselectfor the cases where ECMAScript and C99 differ. On a page module it is a call tostd/math.ts'spowF64, copied in liketan, which answers ECMAScript's cases itself, within one ulp of V8. -
Tests:
math_intrinsics,wasm_math_std;tests/differential/corpus/f64_pow_spec;reject_wasm_math_arity(`Math.pow` expects exactly 2 arguments, got 1).
Math.round(x: f64): f64 is JavaScript's round-half-up, and keeps x's
sign: 2.5 -> 3, -2.5 -> -2, 0.49999999999999994 -> 0, and a
value in [-0.5, 0) or -0 itself -> -0, as in JavaScript; NaN ->
NaN.
-
Lowering:
Math.floor'sfloor+ compare +select, thenllvm.copysign.f64putsx's sign back. - Tests:
math_intrinsics,math_round_negzero,math_rounding.
Math.abs(x: T): T for any numeric T answers the magnitude, wrapping
for INT_MIN. On an unsigned type the value is its own magnitude.
-
Lowering:
llvm.abs.*(wrapping forINT_MIN) /llvm.fabs.f32/llvm.fabs.f64; on an unsigned type, no instruction at all. -
Tests:
math_i32,math_intrinsics;i64,f32and the unsigned widths (CLI only).
Math.min(a: T, b: T): T, Math.max(a: T, b: T): T take exactly two
operands of one numeric type. On floats they give JavaScript's answer: a NaN
operand gives NaN and -0 is below +0 (Math.min(0, -0) is -0).
-
Lowering: signed integers:
llvm.smin/smax; unsigned:llvm.umin/umax. A native float build branches as clang compiles the C a programmer writes for it:fcmp unoto a quiet NaN, elsefcmp olt/ogtand aselect, and onfcmp oeqthe operand whose sign bit is set (min) or clear (max), the three joined by aphi; x86 has no instruction forllvm.minimum, whose expansion measured 25–55% slower inbench/math. A wasm build keepsllvm.minimum/maximum.<f32|f64>, which are wasm'sf64.min/f64.max, so no libm is linked for them. -
Tests:
math_i32,math_intrinsics,math_minmax_float,math_minmax_nan;reject_math_min_arity(`Math.min` expects exactly 2 arguments, got 1).
Math.clz32(x: i32 | u32): integer<0, 32> answers the leading zero bits
of x's 32 bits, JavaScript's answer (an i32 and a u32 are the bits
ToUint32 gives), so 0 answers 32. The answer is ranged
(integer<Lo, Hi>, see Types), so 31 - Math.clz32(word & (0 - word)),
the index of a word's lowest set bit, needs no overflow check. Any other
operand type is refused rather than converted.
- Lowering:
llvm.ctlz.i32withi1 false. -
Tests:
math_clz32;reject_math_clz32_type(`Math.clz32` expects an i32 or a u32, got f64 (convert with toI32 or toU32)).
Math.sign(x: T): T for any numeric T answers -1, 0 or 1 in
x's own type (a ranged x answers its base type). A float -0 and NaN
come back as JavaScript gives them.
-
Lowering: a float answers
llvm.copysign(1.0, x)unlessfcmp ueq x, 0.0, when it answersxitself; a signed integer iszext (x > 0) - zext (x < 0), an unsigned onezext (x != 0). -
Tests:
math_sign;reject_math_sign_type(`Math.sign` expects a number, got string).
Math.imul(a: i32 | u32, b: i32 | u32): i32 answers the low 32 bits of
the product, read signed, with no overflow check, which is JavaScript's
answer for any operands an i32 or a u32 holds
(Math.imul(0x7fffffff, 2) is -2). Any other operand type is refused
rather than converted, for Math.clz32's reason.
- Lowering: one wrapping
mul i32. -
Tests:
math_imul;reject_math_imul_f64(`Math.imul` expects an i32 or a u32, got f64 (convert with toI32 or toU32)).
Math.random(): f64 answers 53 random bits in [0, 1), seeded from
time and pid.
- Lowering: xorshift64* in the runtime (
nish_random). - Tests:
math_random.
Math.PI: f64, Math.E: f64 are constants, also in i32 mode.
- Tests:
math_i32,math_intrinsics.
The f64-only functions reject i32/i64 arguments; number in i32 mode is
an i32, so write Math.sqrt(toF64(n)) or use f64 mode. A numeric literal
argument is typed f64 by context (Math.sqrt(2) works in both modes).
Numeric conversions
The reference for each conversion, with its parameter and its answer, is its doc comment in runtime/nish.d.ts, on the globals page (Number under NumberConstructor); the rules are here.
| Function | Takes | Gives |
|---|---|---|
toI32 |
any number | i32 |
toI64 |
any number | i64 |
toU8, toU16, toU32, toU64 |
any number | that unsigned type |
| same width, other signedness | any integer | the other integer |
toF32 |
any number | f32 |
toF64 |
any number | f64 |
f64ToBits |
f64 |
i64 |
f32ToBits |
f32 |
i32 |
bitsToF64 |
i64 |
f64 |
Number |
string | i32 | i64 | f64 | boolean |
f64 |
parseInt |
string |
i32 |
parseFloat |
string |
f64 |
toI32(x): i32 at narrower or equal width wraps. A float source (f32 or f64) saturates: NaN -> 0, and out of range clamps.
On an i32 it is the identity.
-
Lowering: narrower or equal width:
trunc(wraps); wider signed source:sext, wider unsigned source:zext; a float source: saturatingllvm.fptosi.sat.i32.<f32|f64>;i32: identity. - Tests:
conversions,u_conv_roundtrip;reject_toi32_string.
toI64(x): i64 saturates from an f64, and on an i64 it is the
identity.
-
Lowering: signed source:
sign-extend; unsigned source:zext;f64: saturating;i64: identity. - Tests:
conversions,u_conv_widths.
toU8, toU16, toU32, toU64 widen a signed source by its sign, so
toU64(-1: i32) is 2^64 - 1, as -1i32 as u64 is in Rust. Narrowing
truncates. A float source saturates, clamped to 0 .. 2^bits-1, so a
negative value is 0.
-
Lowering: widening:
zextfrom an unsigned source,sextfrom a signed one; narrowing:trunc; a float source: saturatingllvm.fptoui.sat, whose clamp is0 .. 2^bits-1. -
Tests:
u_conv_widths,u_conv_f64,u_conv_roundtrip;reject_u_conversion_arg.
any integer -> an integer of the same width and other signedness keeps
the bits as they are (toU32(i) on an i32, toI64(q) on a u64).
- Lowering: no instruction at all: signedness is not part of the LLVM type and the bits do not move.
- Tests:
u_conv_same_width.
toF32(x): f32 rounds an f64 to nearest, and on an f32 it is the
identity.
-
Lowering: signed integers:
sitofp; unsigned integers:uitofp;f64:fptrunc(rounds to nearest);f32: identity. - Tests:
f32_conversions,f32_constant.
toF64(x): f64 widens an f32 exactly, and on an f64 it is the
identity.
-
Lowering: signed integers:
sitofp; unsigned integers:uitofp;f32:fpext(exact);f64: identity. - Tests:
conversions,u_conv_f64,f32_conversions.
f64ToBits(x: f64): i64 reinterprets the 64 bits and never converts the
value: toI64(1.5) is 1, f64ToBits(1.5) is 0x3FF8000000000000.
Under --deterministic a NaN gives
0x7FF8000000000000 (quietBitsF64).
-
Lowering: one
bitcast, so a function that only reinterprets bits staysreadnone. -
Tests:
conv_f64_bits;reject_f64tobits_type,reject_f64tobits_arity.
f32ToBits(x: f32): i32 is the same for an f32's 32 bits, signed as an
i32: f32ToBits(1.5) is 0x3FC00000, and f32ToBits(-0) is
-2147483648. Under --deterministic a NaN gives
0x7FC00000 (quietBitsF32).
-
Lowering: one
bitcast float to i32; under Node,runtime/nish.mjssupplies it from aDataView. -
Tests:
f32_to_bits;reject_f32_to_bits_f64(`f32ToBits` expects an argument of type f32, got f64).
bitsToF64(b: i64): f64 is the inverse. bitsToF64(f64ToBits(x)) === x
for every x except NaN, which is not === to itself.
- Tests:
conv_f64_bits;reject_bitstof64_type.
Number(x: string | i32 | i64 | f64 | boolean): f64 parses a string with
JavaScript's Number(s) rules (below). A boolean converts with true as
1, and an f64 is the identity. A numeric literal argument is typed f64
(Number(2.5) works in i32 mode). Under
--deterministic a string goes to std/math.ts's
numberExact instead, which reads a 0x integer the same on every target.
- Lowering:
i32/i64:sitofp;boolean:uitofp;f64: identity. -
Tests:
parse_numbers;reject_number_array(`Number` expects a string, number, or boolean, got i32[]).
parseInt(s: string): i32 reads base 10 only: ASCII whitespace, an
optional sign, then digits; it stops at the first other character.
Deviations from JavaScript: no NaN in an i32, so no digits give 0
(parseInt("abc"), parseInt("")); values beyond the i32 range saturate
exactly like toI32 (parseInt("99999999999") is 2147483647); a 0x
prefix is not hexadecimal (parseInt("0x10") is 0). It is always i32,
also under --number-mode f64 (use Number there).
- Lowering:
strtoll. -
Tests:
parse_numbers;reject_parseint_number(`parseInt` expects a string, got i32).
parseFloat(s: string): f64 reads ASCII whitespace, then the longest
decimal literal ([sign] digits [. digits] [e [sign] digits], . digits,
or [sign] Infinity), correctly rounded. It answers NaN when there is none
(parseFloat("abc"), "", "."); parseFloat("3.14xyz") is 3.14,
"1e" is 1. Deviation: a 0x prefix is read as hexadecimal like
strtod (parseFloat("0x1A") is 26; JavaScript stops at the x and
gives 0). Under --deterministic a string goes to
std/math.ts's parseFloatExact instead, which reads a 0x integer the
same on every target.
- Lowering:
strtod. - Tests:
parse_numbers.
The string parsers accept only ASCII whitespace ( \t\n\v\f\r), not the
Unicode spaces JavaScript also trims. Number(s) trims it on both sides and
then requires the whole string to be one literal: Number("") and
Number(" ") are 0, Number(" 7.5 ") is 7.5, Number("12px"),
Number("1e"), Number("Infinityx"), Number("nan") and Number("inf")
are NaN, Number("0x1A") is 26 (as in JavaScript; hex floats such as
0x1p3 are also accepted by strtod, which JavaScript rejects), and the
0b/0o prefixes are NaN (JavaScript reads them). All three lower to
nish_parse_number(s, mode) in the runtime (mode 0 parseFloat, 1 Number,
2 parseInt, the last followed by llvm.fptosi.sat.i32.f64); their effect
is write (strtod/strtoll may set errno), so a function that parses is
never readonly (wp7-runtime.md).
nish:unsafe: unchecked access and defined wrapping
Two things the language otherwise never does are each a decision about one
site: reading or writing an element without the bounds check, and letting a
signed + - * wrap. nish:unsafe holds the five functions that make them,
two, readCString and readStruct, that read a C string or a C struct
through a pointer nothing proves, and an eighth, cErrno, that reads the
errno a C call left. A program reaches them only through an import, so every module that
gives up a check or defines a wrap says so at its top and every site that does
is a call a reader can find.
Reference: nish:unsafe.
| Signature | Lowering | Effect | Test |
|---|---|---|---|
uncheckedGet(xs: T[], i: i32): T |
the GEP and load xs[i] lowers to, with no len load, no compare and no branch to the bounds panic |
read | unsafe_get_set; reject_unsafe_arity, reject_unsafe_type |
uncheckedSet(xs: T[], i: i32, v: T): void |
the GEP and store xs[i] = v lowers to, likewise unchecked; a statement |
write | unsafe_get_set; reject_unsafe_arity, reject_unsafe_type, reject_unsafe_par_shared_write |
wrappingAdd(a: T, b: T): T |
add, never nsw and never checked |
none | unsafe_wrapping; reject_unsafe_arity, reject_unsafe_type |
wrappingSub(a: T, b: T): T |
sub, never nsw and never checked |
none | unsafe_wrapping; reject_unsafe_type |
wrappingMul(a: T, b: T): T |
mul, never nsw and never checked |
none | unsafe_wrapping; reject_unsafe_type |
readCString(p: CPtr): string |
a NULL test that panics ffi-null, then a call to @nish.ffi.str, the scan-and-copy a string result of a declare function uses (readCString) |
read; allocates; may panic | ffi_owned_string, panics_ffi_read_cstring_null; reject_unsafe_read_cstring, reject_ffi_owned_string_arena |
readStruct(p: CPtr): T |
a NULL test that panics ffi-null, then nish_alloc_struct of the layout's size and one llvm.memcpy into it, T the declare class the context names (readStruct) |
read; allocates; may panic | ffi_struct_read, panics_ffi_read_struct_null; reject_unsafe_read_struct_context, _null, _type_args |
cErrno(): i32 |
a load i32 of @nish_errno_slot; naming it brackets every declare function call with a zero of errno and a save into the slot (cErrno) |
read | ffi_errno, ffi_errno_threads; reject_ffi_errno_arg, reject_ffi_errno_import, reject_ffi_errno_wasm, reject_ffi_errno_global |
-
The import is required. The seven are declared as globals in
runtime/nish.d.ts, as everynish:export is, sotscreads either spelling; the compiler refuses the global one with`uncheckedGet` is a `nish:unsafe` function and is called only through an import of it(NL2456,reject_unsafe_no_import). An import may rename (as) and cannot be shadowed, as every builtin import (above). A user function of the same name, with no import, is the user's function, as with every builtin. -
uncheckedGetanduncheckedSettake an array of numbers:i32,i64,u8,u16,u32,u64,f32orf64elements, and nothing else is`uncheckedGet` takes an array of numbers (i32, i64, u8, u16, u32, u64, f32 or f64), got string[](NL2455). A number is copied in and out and held by nothing, so the store needs none of the escape and inline-array reasoningxs[i] = vcarries. The index is ani32, and anf64one is`uncheckedGet` expects an argument of type i32, got f64; the value of a store is the element type. Areadonlyarray may be read and not written (Cannot `uncheckedSet` through readonly u8[]). An index outside[0, xs.length)is undefined behaviour, which is the whole of the trade. The receiver is an argument, so an array field handed to either is a value and is not stored inline (Fixed-length array fields are stored inline); a store is a write of the receiver for every rule that asks, the parallel-body rule included (reject_unsafe_par_shared_write). -
wrappingAdd,wrappingSubandwrappingMultake two operands of one type,i32ori64, which is also the answer: the signed words are the ones whose operators are checked, and an unsigned operator already wraps.`wrappingMul` takes i32 or i64 operands (convert with toI32 or toI64), got u32is NL2454, and two widths are`wrappingAdd` needs every operand of one type, got i32 and i64(NL2400, asctSelect's). A bare literal takes the other operand's type, wherever it stands. The answer is the two's-complement wrap at that width, defined for every pair (unsafe_wrapping). In a module constant's initialiser the call folds, wrapping, where2147483647 + 1is refused (Module constants). -
Under Node
runtime/nish.mjsanswers the import, and the five arexs[i],xs[i] = v,(a + b) | 0,(a - b) | 0andMath.imulon ani32, andBigInt.asIntN(64, …)on the BigInt ani64is there; a program that stays in range prints the same as natively (unsafe_get_setandunsafe_wrappingare run both ways bytests/run.js).readCStringandreadStructhave no Node form: aCPtrcomes only from adeclare function, which Node cannot call, so a program that reads one is native only, as every FFI program is outside the differential oracle (docs/wp27-ffi.md§5). -
Every import and every call is a recorded fact.
--emit-checkedprints, per module, anunsafe import <line>:<col> <name> from "nish:unsafe"line for each name imported and anunsafe call <line>:<col> <name> (node <id>)line for each site, so a report of where a program opts out of a check reads them rather than the source (tests/self/goldens/checked/, one file per module). -
The import is the opt-in the no-panic scope takes. Under
--deny-panicsor"nish".noPanic,uncheckedGetanduncheckedSetare allowed, listed asuncheckedsites with"allowed": true, because they cannot panic and the import has already said, at the top of the module, that an index out of range is the author's to rule out (The no-panic scope,deny_panics_unsafe).readCStringandreadStructare not: each tests its pointer forNULL, anffi-nullsite no proof removes (panics_ffi_read_cstring_null,tests/ffi-string.js;panics_ffi_read_struct_null,tests/ffi-struct.js), so--deny-panicsrefuses a call to either in its scope. Thewrapping*three are not sites at all: they are defined for every pair. -
To the bounds proof
uncheckedGetanduncheckedSetare the load and the storexs[i]andxs[i] = vare, less the check: a call moves no length, so inuncheckedGet(a, at[0]) + b[at[1]]withatan array of two,at[1]stays proven after the call. Neither records a passed check, because neither has one: a checkedxs[i]after either keeps its check, and a resize in the value stored still takes the length away (tests/cases/arr_bounds_unchecked_kept). -
There is no flag that turns the checks off.
--unchecked-indexingand--wrappingwere removed before 1.0, and the CLI refuses each with exit 2 and a line naming its replacement (`--wrapping` is gone,tests/nish/cli.ts): an opt-out is written at the site, where--emit-checkedand the no-panic scope can see it.
nish:simd: sixteen-byte vectors
A vector is sixteen bytes of lanes of one number type, held and passed as one
value: an LLVM <N x T> in a register, a local, a field or an array slot. The
module names six of them, and the calls that make one, read one out of an
array, write one back and take one lane out or put one in. Nothing here names a
target: the IR is insertelement, shufflevector, extractelement and a
vector load or store, and LLVM picks the instructions (tests/run.js's
simd section greps every simd_*.ll for llvm.x86. and llvm.aarch64.).
The lane operations, the masks a compare answers and the shifts are in
the next section.
The six are u8x16 (<16 x i8>), u16x8 (<8 x i16>), i32x4 and u32x4
(<4 x i32>), f32x4 (<4 x float>) and f64x2 (<2 x double>). Each has
a constructor of one argument per lane, a splat (splatU8x16 … splatF64x2)
and a load (loadU8x16 … loadF64x2) from an array of its lane type; the
table below spells each family with i32x4.
Reference: nish:simd.
| Function | Takes | Gives |
|---|---|---|
i32x4 |
one i32 per lane |
i32x4 |
splatI32x4 |
one i32 |
i32x4 |
loadI32x4 |
readonly i32[], i32 |
i32x4 |
store |
i32x4, i32[], i32 |
void |
extract |
i32x4, constant k |
one lane |
replace |
i32x4, constant k, lane |
i32x4 |
i32x4(a, b, c, d), one argument per lane makes a vector of its
arguments.
-
Lowering: one
insertelementper lane intoundef, whichoptfolds to a constant vector when every lane is one. - Tests:
simd_lanes;reject_simd_arity,reject_simd_lane_type.
splatI32x4(x) makes a vector with x in every lane.
-
Lowering: one
insertelementinto lane 0 and ashufflevectorwhose mask iszeroinitializer. - Tests:
simd_lanes.
loadI32x4(xs, i) reads a vector out of xs at i, after one bounds
check for the whole vector, i < len and i + N <= len.
-
Lowering: the bounds check branches to
nish_panic_index(i, len); then the element's address bitcast to<N x T>*and oneloadaligned to the lane, with the!alias.scope,!noaliasand!tbaaof a scalar element access of the lane type. The check is left out where the bounds proof (src/bounds.ts,judgeVectorAccess) already knows every lane is in range:inon-negative under a guard such asi <= n - Nori + N - 1 < non the array's length, or a literal index into an array of known length. A guard one lane short proves nothing, and the access keeps its check. -
Tests:
simd_access,simd_access_proven,simd_access_alias,simd_access_oob,simd_access_oob_first,simd_access_oob_negative,simd_access_oob_guard;reject_simd_lane_type.
store(v, xs, i) writes v back into xs at i after the same check.
It is a statement.
- Lowering: the same check, then one
store. -
Tests:
simd_access,simd_store_oob;reject_simd_lane_type,reject_simd_arity.
extract(v, k) takes lane k out of v; k is a constant.
- Lowering:
extractelementat the constantk. - Tests:
simd_lanes;reject_simd_lane_index.
replace(v, k, x) puts x in lane k, a constant, and answers a new
vector: v is unchanged.
-
Lowering:
insertelementat the constantk. -
Tests:
simd_lanes,simd_values;reject_simd_lane_index. -
The import is required. The six types and every call are declared in
runtime/nish.d.tsunderdeclare module "nish:simd"only, and a module that spells one without the import is refused with`i32x4` is a `nish:simd` name and is used only through an import of it: add `import { i32x4 } from "nish:simd"`(NL2578,reject_simd_no_import). An import may rename (as) and cannot be shadowed, as every builtin import. -
A load or store takes an array of the lane type, exactly: a
u8x16is loaded from au8[]and nothing else, so the sixteen bytes it reads are the sixteen elements the index names, and anf64[]handed toloadF32x4is`loadF32x4` takes an array whose elements are the vector's lane type, f32[], got f64[](NL2579). Areadonlyarray may be loaded from and not stored to (Cannot `store` through readonly u8[]). The index is ani32; the check is one compare pair for the whole vector, so a load whose last lane is past the end panics before it reads, naming a lane past the end (the first when it is out of range, else the last) and the length, asxs[i]does (simd_access_oob, run bytests/run.jsnatively and under Node). The access is aligned to the lane, not to sixteen: an array's elements are where the arena put them. -
A lane index is an integer literal in range.
extract(v, k)andreplace(v, k, x)takekas a literal from 0 to one less than the lane count, because a constant lane is one instruction and needs no check at run time: a variable is`extract` takes a lane index that is an integer literal, from 0 to 3: i32x4 has 4 lanes(NL2580) and a literal past the end is`extract` lane index 4 is out of range for i32x4, whose lanes are 0 to 3(NL2583,reject_simd_lane_index). The first argument must be a vector (`extract` takes a `nish:simd` vector as its first argument, got i32, NL2582), and a lane given to a constructor, a splat orreplaceis the lane type, as any argument is (reject_simd_lane_type). -
A vector is a value. It is copied by assignment, passed and returned in registers, and held in a local, a class field or an array (
simd_values). In an object or an array it is sixteen bytes aligned to sixteen, the layout clang gives__attribute__((vector_size(16))), whichtests/layout/simd.casserts against the IR's struct and reads lane by lane; an array of vectors steps by sixteen. The arena hands out objects aligned to eight, so a field access isalign 8. -
A vector has no operators, and no comparison.
+, unary-, a truthiness test, a template hole andconsole.logof a whole vector or a mask are refused with the usual operand-type diagnostics (reject_simd_operators); the lane-wise forms are the functions of the next section.===,!==,<and the rest, and the===inside an array'sindexOf, are`===` does not compare `nish:simd` vectors or masks (got i32x4): compare lane by lane with `eq`, `lt` and the rest, which answer a mask, and ask it `all` or `any`(NL2581,reject_simd_compare): a lane-wise compare answers a mask, not a boolean, and under Node a vector is a typed array whose===is identity. A vector is not aMapkey (reject_simd_map_key), notSecret(reject_simd_secret) and not nullable (reject_simd_nullable). -
Under Node
runtime/nish.mjsresolves the import toruntime/core.mjs, where a vector is a typed array whose element type does the lane's arithmetic: au8lane wraps throughUint8Array, anf32lane rounds throughFloat32Array.--emit-tswrites the import asnish/runtime's. A load or a store checks the whole vector and panics with the native words.tests/run.js'ssimdsection runs everysimd_*program with a.outnatively, under--profile wasi(where a sysroot is installed) and under Node, and holds the three to one output. -
A vector does not cross the interop boundary yet. An exported function whose parameter or result holds a vector at any depth (the vector itself, an array of them, a
Resultwith a vector arm, an array of a class that holds one) is left out of--emit-header,--emit-napiand the--emit-dtstypings and wasm loader, with a comment naming the type, as every type the sidecars cannot spell is. A class that holds a vector anywhere (a field, an element, aResultarm, another class it points at) is opaque in the header (typedef struct Name Name;), as aMaporSecretholder is: C has no portable vector type to lay it out with. A bare pointer to such a class still crosses, as an opaquestruct Name *.
nish:simd: lane operations and masks
Every operation on a vector is a free function of nish:simd, because a
TypeScript program cannot overload +. The operand types pick the lowering,
and each lowers to LLVM vector IR or a target-independent llvm.* intrinsic:
no golden holds llvm.x86. or llvm.aarch64. (wp38-simd.md
§3.2).
Reference: nish:simd.
| Signature | Lowering | Test |
|---|---|---|
add, sub, mul, div(a, b) on f32x4, f64x2 |
fadd, fsub, fmul, fdiv, no fast-math flags |
simd_ops_float; reject_simd_op_operands |
neg, abs, sqrt(a) on f32x4, f64x2 |
fneg, llvm.fabs, llvm.sqrt |
simd_ops_float; reject_simd_op_operands |
addWrapping, subWrapping, mulWrapping(a, b) on u8x16, u16x8, i32x4, u32x4 |
add, sub, mul with no nsw or nuw: each lane wraps |
simd_ops_int; reject_simd_op_operands |
shl(v, n), shr(v, n), n a literal below the lane width |
shl; shr is lshr on an unsigned lane and ashr on i32x4 |
simd_shifts; reject_simd_shift_count |
rotl(v, n), rotr(v, n) on the same four |
llvm.fshl, llvm.fshr with v as both halves |
simd_shifts, simd_chacha_quarter; reject_simd_shift_count |
and, or, xor(a, b), not(a) on an integer vector or a mask |
and, or, xor; not is xor with all-ones |
simd_ops_int |
min, max(a, b) on every vector |
llvm.umin/smin/umax/smax; floats llvm.minimum/llvm.maximum, NaN-propagating with -0 below +0, as Math.min |
simd_ops_int, simd_ops_float |
eq, ne, lt, le, gt, ge(a, b) |
icmp (unsigned on u* lanes) or ordered fcmp (une for ne), answering the lane width's mask |
simd_ops_int, simd_ops_float; reject_simd_op_operands |
select(m, a, b) |
select: lane k of a where lane k of m is set, else of b |
simd_ops_int; reject_simd_op_operands |
any(m), all(m) |
llvm.vector.reduce.or, .and on <N x i1> |
simd_ops_int |
bitmask(m), an i32 with bit k lane k |
bitcast <N x i1> to iN, zext |
simd_ops_int |
sum(v) on u8x16, u16x8 (an i32, exact) and f32x4, f64x2 (the lane type, lanes added 0 to N-1) |
zext and llvm.vector.reduce.add; floats llvm.vector.reduce.fadd without reassoc |
simd_ops_int, simd_ops_float; reject_simd_sum |
sumWrapping(v) on i32x4 (an i32) and u32x4 (a u32) |
llvm.vector.reduce.add, wrapping |
simd_ops_int; reject_simd_sum |
shuffle(a, b, i0, …), one literal index per lane, from 0 to 2N-1 |
shufflevector |
simd_ops_int; reject_simd_shuffle |
asU8x16(v) … asF64x2(v) |
bitcast: the same sixteen bytes read as other lanes |
simd_ops_int |
-
Every operand is one vector type. Two types are
`min` takes operands of one type, got i32x4 and u32x4; reinterpret the lanes with `asU8x16` and the rest, a scalar where a vector goes is`eq` takes a `nish:simd` vector, got i32(NL2638), and a vector a family does not take names the family:`add` is float lane arithmetic and takes an `f32x4` or an `f64x2`, got i32x4: an integer lane can overflow, so its form is `addWrapping`(NL2623,reject_simd_op_operands). Integer lane arithmetic wraps, and its name says so: making a lane trap as scalar+does would cost the speed the type is for. -
A shift count is a literal below the lane width.
shl,shr,rotlandrotrtake the count as an integer literal from 0 to 7, 15 or 31, so each is one instruction with a splat constant and nothing is shifted by a variable: a variable count is`shl` takes a count that is an integer literal below the lane width, from 0 to 31: a lane of u32x4 is 32 bits(NL2630), and the width or more is`rotl` count 32 is not below the lane width of u32x4, 32 bits: a shift by the width or more moves every bit out, and a rotate by it is a rotate by 0(NL2631,reject_simd_shift_count). LLVM gives a shift by the width no meaning, and a rotate by it is a rotate by 0.shris logical onu8x16,u16x8andu32x4and arithmetic (it copies the sign) oni32x4; a rotate is the same on either. They are what ChaCha20, SHA-256 and other add-rotate-xor code is built from (simd_chacha_quarter). -
A mask is a compare's answer, held only as a value.
eqand the rest on two vectors answermask8x16,mask16x8,mask32x4ormask64x2by lane width, an<N x i1>; there is no constructor (`mask32x4` is a mask type and has no constructor: a mask is what a lane-wise compare (`eq`, `lt` and the rest) answers, NL2621).selecttakes the mask of its vectors' width (`select` between two i32x4 takes the mask their compare answers, mask32x4, got mask64x2, NL2629). A mask lives in a local, a parameter or the return of a function the module keeps, and is`mask32x4` cannot be a field: a `nish:simd` mask is a value held only in a local, a parameter or a return of a function this module keeps; store the vectors it selects, or its `bitmask`(NL2622) as a field, an array element, aResultarm, a type argument, a method's parameter or result, or an exported signature (reject_simd_mask_storage,reject_simd_mask_in_body): it has no size in memory that all three targets share. -
A sum says whether it wraps.
sumofu8x16oru16x8adds in ani32and cannot overflow;sumofi32x4oru32x4can, and is`sum` of i32x4 can overflow its lane type, so its total is `sumWrapping`, which wraps as the lanes do(NL2636,reject_simd_sum). A float sum adds lane 0 first and lane N-1 last, so it is the same bits on every target. -
A shuffle names every lane.
shuffle(a, b, …)takes one literal per lane of the answer, each from 0 to 2N-1, where N and up areb's lanes; a wrong count, a variable or an index past the end is refused (NL2632 to NL2635,reject_simd_shuffle). -
Under Node each operation is a function of
runtime/core.mjsover the typed arrays: an integer lane wraps through its array type, a float lane rounds throughFloat32Array, a mask is an array of booleans, and a rotate is two shifts and anor.tests/run.js'ssimdsection holds the native, WASI and Node outputs to one.
Constant time: ctSelect and ctEq
Crypto code must not branch on a secret or index memory by one, and LLVM
promises neither: a select it can see through may become a branch, and a mask
it can prove is a boolean may become a select. These two builtins are how a
program selects and compares on a secret without giving the optimiser that
choice (WP34 N6).
| Function | Takes | Gives | Effect |
|---|---|---|---|
ctSelect |
T, T, T |
T |
none |
ctEq |
T, T |
T |
none |
ctSelect(mask: T, a: T, b: T): T answers (a & mask) | (b & ~mask):
a when mask is all-ones, b when it is zero.
-
Tests:
ct_select_u32,ct_u64;reject_ct_select_i32,reject_ct_select_i32_literal,reject_ct_select_f64,reject_ct_select_f32,reject_ct_select_bool_mask,reject_ct_select_mixed,reject_ct_select_arity,reject_ct_select_float_mask.
ctEq(a: T, b: T): T answers all-ones of T when a === b, zero
otherwise.
-
Tests:
ct_eq_u32,ct_u64;reject_ct_eq_i64,reject_ct_eq_u8,reject_ct_eq_f64,reject_ct_eq_f32,reject_ct_eq_bool,reject_ct_eq_ranged,reject_ct_eq_mixed,reject_ct_eq_arity,reject_ct_eq_literals,reject_ct_eq_value. -
Tisu32oru64, one type for every operand. A mask is a bit pattern, and the unsigned words are the only types whose all-ones is not also a number with a sign:i32,i64, the narrowu8andu16, a ranged integer (ani32underneath, and named so),f32,f64andbooleanare each`ctEq` takes u32 or u64 operands (convert with toU32 or toU64), got i64(NL2399). A boolean in particular is not a mask:ctEqmakes the mask without the compare a boolean would need. Operands of two types are`ctSelect` needs every operand of one type, got u32 and u64(NL2400), since au32mask overu64words would leave half of each unselected. The type is judged at the operand it was read from before any other operand is measured against it, so a boolean mask is told the first message rather than the second, andctSelect(m, 0x80000000, 0)on ani32mask is NL2399 rather than "does not fit in i32" (reject_ct_select_i32_literal). -
A bare numeric literal takes its type from the first operand that is not one, wherever it stands:
ctEq(0, x)andctSelect(m, a, 0)are both words ofx's andm's type. A fraction is then refused as a fraction (Non-integer literal `0.5` where u32 is expected,reject_ct_select_float_mask), and with nothing but literals there is no width to take, soctEq(1, 2)is ani32and refused (reject_ct_eq_literals). -
The mask convention is all-ones or zero, and it is not checked, because the check would be the branch.
ctEqonly ever answers one of the two. Any other mask is a defined bitwise blend, bit by bit as the formula says —ctSelect(0xf0f0f0f0, a, b)takes the high nibble of each byte froma— and the Node reading below computes the same blend (ct_select_u32). -
They are globals, not a
nish:module, by the rule Builtin modules sets: what is backed by the C runtime may be imported, and what lowers to instructions reads as language. These call nothing (reject_ct_nish_module). Like every function, neither is a value (reject_ct_eq_value). -
The lowering is the formula behind an optimisation barrier. The barrier is an empty inline
asmthat hands a value back in the same register —call i32 asm "", "=r,0"(i32 %mask) readnone nounwind— which costs no instruction and which no pass can see into, the idiom BoringSSL and Rust'ssubtlecrate use.ctSelectputs it on the mask before the blend;ctEqcomputes((d | -d) >>> (w - 1)) - 1fromd = a ^ band puts it on the answer, so a caller'sctSelect(ctEq(a, b), x, y)never shows LLVM a boolean.readnone nounwindis what clang writes on the same statement and is true of an empty string, so a function that only selects staysreadnone. IR_COOKBOOK.md shows the IR. Without the barrieropt -O2foldsctSelect(ctEq(a, b), x, y)into oneselectona === b—tests/run.jschecks both halves of that sentence, so it learns if LLVM ever sees through theasmor stops needing it. -
What the barrier promises: no branch and no secret-indexed load in these primitives, checked where the promise lives, in the machine code.
tests/run.jscompiles everytests/cases/ct_asm_*fixture withclang -O2 -Sfor x86-64 and aarch64 and reads each function a fixture names by its symbol (tests/ct-asm.js). It refuses any conditional branch (j<cc>,b.<cond>,cbz,tbzand the rest), any call or jump out of the function, and any load or store whose address a secret reaches, following secrets through registers, flags and stack slots and erring towards refusal wherever its model of the machine is not exact. The fixtures are both builtins at both widths, a mask straight into a select, a 32-byte MAC compare overu8[]and a table read that loads every entry (ct_asm_primitives,ct_asm_mac); andct_asm_refusedholds an early-exit compare and a secret-indexed table lookup that the check must refuse, one for each kind, so a check that read nothing fails. -
It also refuses an instruction whose latency follows a secret: an integer divide (
div/idiv,udiv/sdiv) or a floating divide or square root whose operand a secret reaches (thelatencykind, seeded byct_asm_refused'ssecretDivide). It reads whatclang -O2emits for the baseline CPU of each target, and for x86-64 atx86-64-v3, the one level--cpuoffers, and nothing else, so a build with another profile or-marchis not what was read (docs/security/ct-verification.md, CT-2, CT-16). It reads only the functions a fixture names, so a module function that is not one — every loop instd/crypto, for instance — rests on review; that record lists them. -
What it does not promise: anything about the caller's own code.
if (ctEq(a, b) !== 0)branches on a secret because the program wrote a branch;table[secret]indexes by one;===on a secret is a compare LLVM may branch on; a loop whose trip count is secret leaks it; and/and%take time that depends on their operands on many processors. The fixtures show the shapes that stay constant time, and a new one belongs beside them. Onwasm32the builtins compile with the same barrier, but the engine compiles the module again, so the.sthis check could read is not what runs and nothing is promised there. Weekly, and never fromnpm test,.github/workflows/ct-timing.ymlrunstests/ct-timing.json an x86-64 and an aarch64 runner — a dudect-style Welch's t-test of every function the fixtures name, a fixed secret against a random one, filed as an issue when |t| passes 4.5 — which measures whether their running time on that hardware moves with the secret, and cannot promise more: a difference below a shared runner's noise, a secret index into a table that fits in the L1 cache, or another processor goes unseen.
The TypeScript reading (wp33-round-trip.md §2):
runtime/nish.mjs supplies both as pure functions of their operands, so the
answers agree and only the timing is not claimed. Over u32 they are
class A: a u32 is a number there, the shim puts each answer back in
range with >>> 0 (JavaScript's & answers a signed 32-bit integer), and
ct_select_u32 and ct_eq_u32 run unmodified under the prelude and print what
the native build prints. Over u64 the prelude takes BigInts and answers
BigInts, and the ct_prelude check in tests/run.js holds its answers to
ct_u64's native output line for line. A u64 program is still class C
under Node, as all 64-bit arithmetic is (RUN_UNDER_NODE.md):
a bare u64 literal is a number there, and mixing one with a BigInt throws a
TypeError rather than comparing the two and answering zero.
Wiping a secret: secureZero
A key, a secret scalar or another secret intermediate has to be cleared once it
is used, and a store the program never reads again is exactly what the optimiser
removes: key.fill(0) just before a key goes out of scope is a dead store at
-O2, and a memset in the runtime is one too once -flto inlines it into
that caller. This builtin is the store that stays (#385).
| Signature | Semantics | Effect | Test |
|---|---|---|---|
secureZero(bytes: u8[]): void |
every byte of bytes set to zero, in stores no optimisation removes; an empty array is a call that clears nothing; a statement |
write | wipe_bytes; reject_wipe_i32, reject_wipe_string, reject_wipe_readonly, reject_wipe_value, reject_wipe_arity, reject_wipe_region |
-
A
u8[]only, ascrypto.getRandomValuestakes, because the byte is the unit a secret is written in: anything else is`secureZero` fills a `u8[]`, got i32[](NL2402), a string included, which is immutable and has nothing to clear (reject_wipe_i32,reject_wipe_string). The call writes its argument, so areadonly u8[]isCannot `secureZero` through readonly u8[] (declare it u8[] to write through it)(NL2264,reject_wipe_readonly), and a function that wipes a parameter does not getreadonlyon it (wipe_bytes). In a scope's region it is refused wherefillis (NL2394,reject_wipe_region). -
A statement: there is nothing to answer, so
`secureZero` returns void and can only be used as a statement(NL2105,reject_wipe_value), and it takes exactly one argument (NL2061,reject_wipe_arity). -
The lowering is one call,
call void @nish_wipe(%struct.nish_array* %key), declarednounwind willreturnandnocapturewithoutreadonlyand with no memory attribute, so LLVM may not treat it as a store it can drop.nish_wipeinruntime/runtime.cwrites each byte through avolatilepointer, which the C standard makes behaviour no optimisation may remove even once LTO has inlined the body, and isnoinlinebesides, so the call stays visible in a disassembly. IR_COOKBOOK.md shows the IR. -
It is pinned under
-O2 -flto, where it matters. The runtime unit test builds a probe the way--linkbuilds,-O2 -fltoover the runtime's sources: a key written to a block, used, wiped and freed, then read back through a pointer the optimiser cannot follow. With no wipe the key is still there, and with amemsetwipe in another translation unit it is still there too, dropped as a store beforefree; withnish_wipenothing is left. The same test fails whennish_wipeis written as amemset. -
What it does not promise: the copies the program made. It clears the
bytes of the array it is handed. A copy taken before (
slice, apushinto another array, an element read into a local, a register spilled to the stack) is the program's to clear, and nothing clears what the operating system swapped out. -
It is a global, and not in a
nish:module. It is backed by the C runtime, so by the rule Builtin modules sets it may be imported, but no module exports it yet. It works under the wasi profile, which linksruntime.c; a freestanding wasm module has nonish_wipe: it linksruntime-wasm.c, which supplies the arena, arrays, the panics andfloor/ceil/truncbut no wipe, and, when it uses strings,runtime-text.c, which has the strings and the digits but not the wipe.
The TypeScript reading (wp33-round-trip.md §2) is
class A: runtime/nish.mjs supplies it as a loop that sets every element
to zero, and wipe_bytes prints the same lines under the prelude as natively.
What Node cannot promise is the native build's stronger claim, because a
collector that moved the array may have left the old bytes behind.
Who owns a path: lstatOwnerModeSync, geteuid and isExecutableSync
What a driver asks before it trusts a directory or a program it found on its own: who owns this path, may anyone else write to it, and will the shell run it (#386; security/cli.md, CLI-7 and CLI-9).
| Function | Takes | Gives | Effect |
|---|---|---|---|
lstatOwnerModeSync |
string |
i64 |
write |
geteuid |
nothing | i64 |
write |
isExecutableSync |
string |
boolean |
write |
lstatOwnerModeSync(path: string): i64 does one lstat of path. It
answers the owner's uid in the high 32 bits and st_mode (the file type and
the permission bits) in the low 32, or -1 when the path does not resolve
or holds a NUL. A symbolic link answers for itself, not for what it
names.
-
Tests:
owner_checks;reject_owner_arity,reject_owner_i32,reject_owner_wasm.
geteuid(): i64 answers the effective user id, the one to compare an
owner with.
-
Tests:
owner_checks;reject_owner_euid_arity,reject_owner_euid_wasm.
isExecutableSync(path: string): boolean is access(path, X_OK):
whether the real user may run path. It answers false for a path that
does not resolve or holds a NUL.
-
Tests:
owner_checks;reject_owner_type,reject_owner_exec_wasm. -
The owner and the mode are one answer. Both come out of one
lstat, so they describe the same file however the path changes in between; two calls could describe two. Take them apart with(om >> 32) & 0xffffffff(the owner) andom & 0xfff(the permission bits) orom & 0xf000(the type:0x8000a regular file,0x4000a directory,0xa000a symbolic link). The mask is not optional: a uid is unsigned and one of 2^31 or more fills bit 63, which>>on ani64copies down, soom >> 32alone answers a negative owner for a file such a user owns and never equals theirgeteuid().>>>would say the same natively, but a BigInt has no>>>, so the mask is the spelling that reads the same under Node (owner_checks). A uid fills 32 bits, so the answer is ani64in both number modes, asgeteuid's is, and ani32is refused rather than truncated (Cannot initialize i32 variable `om` with i64,reject_owner_i32). -
Not following a link is the point. A directory that is a symbolic link to one this user owns is not a directory this user owns, and the test of a path that has to be trusted is of the path as written. That a link answers for itself is pinned by RT-9 in
tests/runtime-test.c, since a program cannot make one. -
isExecutableSyncis the shell's question, asked for the real user asaccessasks it: a file with no execute bit is not executable, root included, and a readable file is not the same answer. -
Refused on a wasm32 build, as
statMtimeSyncis: there is no file system and no user there, so each is`lstatOwnerModeSync` reaches the operating system, and a wasm32 build has none to reach(reject_owner_wasm,reject_owner_euid_wasm,reject_owner_exec_wasm). Each takes exactly its arguments (NL2061,reject_owner_arity,reject_owner_euid_arity), and a path is astring(reject_owner_type). -
The lowering is one call each,
call i64 @nish_lstat_owner_mode(i8* %p),call i64 @nish_euid()andcall zeroext i1 @nish_is_executable(i8* %p), declarednounwind willreturnand ordered with every write, because the file system is not memory LLVM tracks and a failure storeserrno(IR_COOKBOOK.md). -
Globals, and not in a
nish:module yet, assecureZerois.
The TypeScript reading (wp33-round-trip.md §2) is
class A: runtime/nish.mjs supplies fs.lstatSync's uid and mode
packed the same way as a BigInt, process.geteuid(), and fs.accessSync with
X_OK, and owner_checks prints the same lines under the prelude as natively.
Secrets: nish:secret
Key material is a Secret<T>: an opaque value that the checker keeps from
leaving the program by any road but one, and that is zeroed, with a store no
optimiser may remove, before the function that made it lets go of it.
Reference: nish:secret.
| Export | Meaning | Test |
|---|---|---|
Secret<T> |
an opaque class holding an array or record of integers | secret_wipe, secret_flow; reject_secret_shape |
secret(v) |
v, which nothing else holds, wrapped; a local is moved |
secret_flow, secret_inferred; reject_secret_moved, reject_secret_unmovable, reject_secret_alias_*, reject_secret_inferred_* |
expose(s, f) |
f(value), under the rules below |
secret_wipe; reject_secret_expose_* |
exposeWith(s, arg, f) |
f(value, arg), under the same rules |
secret_wipe; reject_secret_expose_writes_arg |
wipe(x) |
zero a Secret, or an array or record of integers expose's function holds |
secret_wipe, secret_wipe_o2, secret_wipe_o2_inferred; reject_secret_wipe_shape, reject_secret_wipe_nullable |
nish:secret is the one builtin module with source behind it: std/secret.ts,
loaded like a standard-library module, imported only by the builtin name
(`nish/secret` is the builtin module `nish:secret`, NL2453,
reject_secret_std_spelling) and copied, as std/collections.ts is, into each
module that uses it, so it writes no .ll of its own. runtime/nish.d.ts
declares it for tsc, with a private constructor and field so that tsc
refuses new Secret and s.value too. Under Node (runtime/nish.mjs) a
Secret is a plain wrapper and wipe zero-fills it.
A Secret local needs no annotation. const k = secret(v) gives k the
type secret answers, Secret<T> for the T that v is, and every rule
below holds for it as for const k: Secret<T> = secret(v): the payload's
shape, the move and the second reference, each judged where v is written
(secret_inferred binds an array literal, new Array, a call, a moved local,
a ternary, a record and a class unannotated; reject_secret_inferred_literal,
_alias, _call, _field and _moved_class are refused unannotated as
annotated). The same holds for any generic an imported module declares:
const b = box(7) holds the Box<i32> that box answers, whose layout comes
with the call (tests/link/generic_import_inferred), with its methods even
where the module already held Box<i32> as a type argument
(tests/link/generic_import_inferred_held), and with those of every struct
its fields reach, through an array, a nullable, a Result or a nested
instance, in either order (tests/link/generic_import_inferred_closure).
What a Secret may hold. One run of integers, so that wipe reaches every
byte with one store: `Secret<string>` is not key material (NL2428,
reject_secret_shape). wipe takes a Secret, never a nullable one (narrow it
first), or an array or record of integers, and anything else is
`wipe` zeroes a `Secret`, or an array or record of integers, and `string` is neither
(NL2438, reject_secret_wipe_shape, reject_secret_wipe_nullable).
Where a Secret may not go. Each place is refused at the point it is
written, before the general rule it would also break, so the one diagnostic
names the secret:
- a template literal hole (NL2431,
reject_secret_template); -
an argument of any builtin —
console.log, a file write,nish:net, a process, an environment read,crypto.getRandomValues— named or imported (`console.log` cannot be handed a `Secret`, NL2432,reject_secret_console,reject_secret_file_write,reject_secret_net_write); -
===and!==against anything but anotherSecretornull(NL2433,reject_secret_equality):k === nullis how a nullable one narrows; - the condition of an
if, a loop or a ternary (NL2434,reject_secret_condition); -
an index, as the receiver or as the index (NL2435,
reject_secret_index,reject_secret_as_index); -
a member read, written or called anywhere but
std/secret.ts(NL2436,reject_secret_member), andnew Secret(NL2437,reject_secret_new); -
an array element, a
Resultpayload, or a type argument of any class,MapandSetincluded (NL2429,reject_secret_array,reject_secret_result,reject_secret_generic_class). ASecretlives in a local, a parameter, or a holder's field (ASecretfield), andSecret<T> | nullis how a function hands back one it may not have. Any other field — of an interface, of a generic class, notreadonly, or not exactlySecret<T>— is NL2430 (reject_secret_field,reject_secret_field_interface,_generic_class,_not_readonly,_nullable).
A Secret leaves the function that made it returned or wiped, on every
path. This is Result rule 1's analysis in its two places — the statement that
drops a value, and the body that never handles one — with the second walked
path by path rather than counted, because "wiped on one arm of an if" is not
"wiped" (reject_secret_one_branch):
-
A
Secreta call makes has an owner: aconstorlet, areturn, orwipe. Anywhere else — a statement, a call's argument — it is`Secret<u8[]>` is made here and nobody owns it(NL2439,reject_secret_discarded,reject_secret_temporary). -
Every
return,break,continueandorReturn()that leaves aSecretthe function made neither wiped nor returned is`k` holds a `Secret<u8[]>` that this exit leaves neither wiped nor returned(NL2440,reject_secret_unwiped_return,reject_secret_break,reject_secret_or_return), and a block that can reach its end with one is NL2441 (reject_secret_scope_end). A branch that proves the localnull(if (k === null)) holds none, apanicorprocess.exitends the process and the key with it, and a loop is walked to its fixpoint, so awipeinside a loop that may run no pass does not count (secret_flow). -
A
Secretlocal may not be assigned while it still holds one (NL2442,reject_secret_overwrite), read once any path has wiped it (NL2443,reject_secret_after_wipe), or copied:const j = kmoveskwhen the function made it, andkis not read again (NL2446,reject_secret_moved_out), but a parameter, an element of an array literal or an arm of a ternary is a second name for one key (NL2444,reject_secret_copy_param). -
secret(v)takes a value nothing else holds, and moves it, because a second name for the bytes would read the key outside theSecret, and zeros oncewipehas run. A local handed in is not read again (NL2445,reject_secret_moved, whose message says the plain array, andreject_secret_inferred_moved_classandreject_secret_moved_interface, where it names the class or interface the plain value is), and neither is the local in either arm ofsecret(c ? a : b). A parameter, a field or an element, whose owner keeps the plain bytes, is refused (NL2447,reject_secret_unmovable). Everything else is judged where it is written:-
an array or object literal,
null, and a ternary of fresh values are fresh, and so is anewwhose constructor keeps nothis; -
a call is fresh when its callee answers a fresh value, judged over the
callee's own body by this same rule, through every depth of calls (a
recursive one included);
exposeandexposeWithanswer what the function they run answers, andr.orReturn(),r.expect(m)andr.unwrapOr(d)whatrcarries in itsOk(dtoo, which may not be a local here, since the move does not reach it). A method of a value,rows.pop(), may answer what the value holds; -
a local is fresh when it is the sole reference to its value: every value
it is given is fresh (another local is not, because that local still
names it), it is not a
for...ofbinding, which names an element, and no other use of it leaves a second name. Reading an integer element or field,.length, an array method on it,===, iterating over it,wipe, and handing it to a builtin, or to a function or method the whole-program facts say keeps nothing of it, are all such uses. An initialiser, an assignment's value, a field or element store, a literal's element, a ternary arm, either side of??, the argument ofOkorErr, an assignment to it used as a value (alias = raw = [1],xs.push(raw = [1]); one whose value is dropped, as a statement or in aforheader, is not), a later declarator of the same statement, areturn, an argument of a method of another value (rows.push(raw),m.set(k, raw)), a read by an arrow that captures it (a data-parallel body or a spawned task), and a function or constructor that keeps it each leave one.
Each is
`secret` takes a value nothing else holds, and `raw` has a second reference: `const alias = raw` (line 6), naming the reference and its line, or the callee and its parameter that keep it (NL2512,reject_secret_alias_const,reject_secret_alias_let,reject_secret_alias_reassigned,reject_secret_alias_assign_chain,_assign_init,_assign_push,_assign_field,_assign_kept,_assign_helper,_assign_parents,_assign_constructor,_assign_source,reject_secret_alias_declarators,reject_secret_alias_field,reject_secret_alias_element,reject_secret_alias_param,reject_secret_alias_return,reject_secret_alias_mutual,reject_secret_alias_ternary,reject_secret_alias_nullish,reject_secret_alias_ok,reject_secret_alias_err,reject_secret_alias_unwrap_or,reject_secret_alias_unwrap_helper,reject_secret_alias_unwrap_local,reject_secret_alias_constructor,reject_secret_alias_receiver,reject_secret_alias_pop,reject_secret_alias_for_of, andreject_secret_alias_capture_map,_from,_map_into,_reduceand_spawn, andreject_secret_alias_spawn_argand_spawn_withfor a task's argument, which the scope keeps until the task has run). Pass a fresh value instead, or a copy:const copy: u8[] = new Array<u8>(raw.length); copy.set(raw), thensecret(copy). A declarator that readsrawaftersecret(raw)in the same statement is a moved read, NL2445 (reject_secret_alias_declarator_moved). The rule refuses some programs that are safe, each with the reference named: a data-parallel arrow that reads the local before the move, aResultlocal copied into another (reject_secret_alias_result_copy), and aletgiven another local even after the move (reject_secret_alias_reassigned_after). -
an array or object literal,
-
A
Secretparameter is borrowed. The callee may read it throughexposeor hand it on, and owes nothing: the caller wipes what it made. It neither returns it, which would give the caller a second name for a key it already owns (NL2546,reject_secret_field_launder_functionand its siblings, among themreject_secret_field_launder_localfor a caller's own local), nor wipes it, which would leave the caller reading zeroes and wiping it again (NL2548,reject_secret_field_wipe_paramand its siblings, among themreject_secret_field_wipe_param_local). The cases are listed under ASecretfield.This narrows 0.18.0. Up to 0.18.0 a callee could also return or wipe a borrowed
Secret; from the release after it, both are refused, for everySecretparameter and not only a holder's field. A function that returned its parameter is called for its effect and the caller goes on using the key it passed; a function that wiped its parameter leaves the wipe to the owner, the function or holder that made the key.
A Secret field
A key that lives as long as an object does — a connection's traffic secret, a
signer's private scalar — is kept in a readonly field of exactly Secret<T>,
in a class with no type parameters. Such a class is a holder. The checker
proves three things of it: the field is set once, from a fresh value; it is
read only where the key cannot get out; and the holder's wipe() ends it.
class Session {
readonly key: Secret<u8[]>;
constructor() {
this.key = secret(new Array<u8>(32)); // once, from a fresh value
}
rekey(by: u8): void {
exposeWith(this.key, by, refill); // rewritten in place, never replaced
}
wipe(): void {
wipe(this.key); // every field, on every path
}
}
-
The releaser. A holder declares
wipe(): void, taking no parameters (`Holder` holds the `Secret` field `k`, so it declares `wipe(): void`, NL2540,reject_secret_field_no_wipe,_wipe_signature). Its body callswipe(this.f)itself for everySecretfield, on every path: the field is walked as an owned local would be, so areturnbefore it, one arm of anif, a loop that may run no pass, and a helper method that wipes it are each`this.k` can leave this `wipe()` unwiped(NL2541,reject_secret_field_wipe_missing,_wipe_return,_wipe_branch,_wipe_loop,_wipe_helper, and_wipe_new, where a constructornewreaches zeroes the key in place). That method is the only place a field ends:wipe(h.f)anywhere else — a function, another method, a constructor, or a function the constructor handsthisto or stores it for — is`k` is wiped outside its holder's `wipe()`(NL2547,reject_secret_field_wipe_outside,_wipe_through,_wipe_method,_wipe_ctor,_this_fin,_this_store); call the holder'swipe(). -
Construction. The field is set in the constructor, by
this.f = secret(v)and nothing else, andvis judged by the sole-reference rule above (NL2512,reject_secret_field_alias_raw). A parameter, a call that answers aSecret, or a local holding one is`this.k` is set from something other than `secret(...)`(NL2542,reject_secret_field_param,_call,_local); a second store while the first may still be there, a later loop pass included, is NL2543 (reject_secret_field_twice,_ctor_loop). The field isreadonly, so a store anywhere else — a method, a function, a compound+=, or a second name forthis— is thereadonlyrefusal, NL2181 (reject_secret_field_store_method,_store_outside,_compound,_this_escape). The store is a statement of its own:const x = (this.k = secret(v))binds the key a second time, and is NL2544 (reject_secret_field_store_value). A holder is made only bynew: an object literal is NL2545 (reject_secret_field_object_literal), and typed JSON neither encodes nor decodes one (NL2485,reject_secret_field_json,_json_decode). -
Reads.
h.fis borrowed, as aSecretparameter is: it is an argument of a call or anew—expose,exposeWith, or a function that takes aSecret, andwipeonly in the holder's ownwipe()(NL2547) — or an operand of===or!==against anotherSecret(a field is nevernull, soh.k === nullis a type error, NL2024). Bound to a local, assigned, returned, chosen by a ternary or??, or read through?.or a chain of fields, it would be a second owner of the key, and is`k` is a `Secret` field, which is read only as a call's argument or an `===`/`!==` operand(NL2544,reject_secret_field_bind,_assign_local,_return,_ternary,_nullish,_optional_chain, and_nestedforo.h.k;_chainis the chained storethis.k = this.j = secret(v)). What a borrow receives stays borrowed: a function, an arrow or a genericid<T>that takes aSecretand returns it would hand the caller a second name for the holder's key, and is`s` is a `Secret` parameter, which its caller owns and wipes: returned(NL2546,reject_secret_field_launder_function,_launder_arrow,_launder_generic,_launder_method, and_launder_localfor a caller's own local); one that wipes it is NL2548 (reject_secret_field_wipe_param,_wipe_param_new,_wipe_param_method, and_wipe_param_localfor a caller's own local). A ternary that picks one of two borrowed parameters to return is refused before it gets there, as a copy (NL2444,reject_secret_field_launder_ternary). Every place aSecretmay not go is closed to the field too:reject_secret_field_console(NL2432),_template(NL2431),_equality(NL2433),_condition(NL2434),_index(NL2435),_member(NL2436),_array,_array_literaland_result(NL2429),_new_arg(NL2439: aSecretmade for a constructor that keeps none, beside an acceptednew Box(h.k)that borrows the field), and_secret_argand_holder_payload(NL2428: neither aSecretnor a holder is key material). -
The holder is a value. It may be aliased and kept in an array, a
Map, another object's field or aResult. The rules that follow are the general ones, applied to a holder as to any class: a data-parallel body may not capture one (NL2504,reject_secret_field_capture) nor may a task (NL2509,_task_wipe), the scope's own thread may not wipe one a task of it may read before the join — throughwipe(), a helper or a constructor (_scope_wipe,_scope_helper,_scope_new) — andusingdoes not take one (NL2387,reject_secret_field_using).
secret_field runs all of it, and tests/run.js holds the wipe to -O2
("nish:secret: a holder's wipe survives opt -O2"): secret_field_wipe_o2's
wipe() is a volatile llvm.memset of the key after opt -O2, where its twin
secret_field_wipe_o2_plain, zeroing the same bytes with fill(0), has an
ordinary one. An exported holder appears in an --emit-header sidecar as an
opaque struct, a typedef with no fields: with its layout a host could
allocate one and skip the constructor that sets the key, so it reaches C only
as a pointer. A Secret an exported function takes or returns is likewise a
pointer to an opaque struct, as a Map is, and --emit-dts and --emit-napi
bridge no function that takes or returns a holder or a Secret
("nish:secret: --emit-header keeps a Secret and its holder opaque, and
--emit-dts / --emit-napi bridge neither").
What the rule leaves out, so a review need not look for it here:
-
A holder nobody wipes. The checker proves that
wipe()ends every key, not that anybody calls it: a holder dropped, or released with its arena, withoutwipe()leaves its key in memory. Holders are values that cross functions, aliases and containers, and their owner is not tracked. -
Aliases.
wipe()through one alias zeroes the key for every alias, and anexposethrough another reads zeros afterwards: a wrong answer, never a leak. - Plain temporaries. What an exposed function copies out of the key is a plain value, and is wiped only if that function wipes it.
-
Per-packet and per-record keys. These cannot be a
Secretunder today'sexpose: sealing writes the output buffer (NL2450) and the AES kernels are C (NL2448).
expose and exposeWith. The function either runs is a
function parameter: a top-level function named at the
call, or an arrow written there. It is judged once the whole-program facts are
in, at the call, and what it returns is the only thing that leaves:
-
It reaches no I/O — no builtin that touches a file, a directory, a process,
the environment, a socket, a stream, the clock or entropy — and no C, through
any depth of calls (
`leak` reaches `writeFileSync` through `store`, NL2448,reject_secret_expose_io,reject_secret_expose_io_deep,reject_secret_expose_c). Apanicor exit is allowed only with a literal, since a computed message or status is output (reject_secret_expose_panic). A failed runtime check — an index, a slice, a division — still stops the program with its message, which carries the operand it failed on; an index that depends on a key is already a cache-timing defect, and is for the constant-time checks of WP34 N6 to find. -
It keeps nothing of the value: returning it, storing it or passing it where
it is kept is NL2449 (
reject_secret_expose_keeps).exposeWith's function writes nothing through the argument it is handed either (NL2450,reject_secret_expose_writes_arg), which would be a road out its result type does not name. -
Its result is a declared type that holds no
Secret(NL2451,reject_secret_expose_secret_result), and an arrow writes that type down (NL2452,reject_secret_expose_undeclared).
The wipe. Each wipe instance is one llvm.memset with its volatile flag
set, over an array's whole capacity — a pop leaves the old element in the
buffer — or a record's sizeof. LLVM never removes, merges or narrows a
volatile memory intrinsic, so the store survives every optimisation, inlining
included: tests/run.js compiles secret_wipe_o2, whose wiped bytes nothing
reads again, beside a twin that zeroes them with fill(0), and after
opt -O2 the twin holds no store and the wipe holds one volatile zero per byte.
secret_wipe_o2_inferred does the same for a Secret bound with no
annotation, beside secret_wipe_o2_inferred_plain ("nish:secret: an inferred
Secret's wipe survives opt -O2 (a volatile memset of its key), where the
twin's fill(0) is deleted").
Two things it does not reach, and says so: a buffer a push outgrew, which is
no longer the array's, and a string, which is immutable and is not key material.
wipe and secureZero both make a store no
optimisation removes, and differ in what they take and where the store is
written. secureZero is a global over a bare u8[], one call into the C
runtime; wipe is the Secret's own, takes any integer width or a record of
integers, and is an intrinsic emitted into the program, so it needs no runtime
function and works wherever the program's IR does. Key material belongs in a
Secret, and secureZero is for bytes that never were one.
Compact classes
class R implements Compact (import { Compact } from "nish:compact")
stores every field typed exactly R or R | null — a link, in R itself
or in any other class — as a 32-bit offset into the one address range the
arena reserves, where a plain class stores a 64-bit pointer. A list or tree
node of an i32 and two links is 12 bytes instead of 24, so twice as many fit
in a cache line.
import { Compact } from "nish:compact"
class Cell implements Compact {
value: i32
next: Cell | null
constructor(value: i32, next: Cell | null) {
this.value = value
this.next = next
}
}
-
Only a link moves. A local, a parameter, a return, an array element, a
Mapkey or value, and a field whose type is a wider union all stay full pointers. A load of a link widens it back to a pointer and a store narrows it; what the program reads and prints is the same as without the marker (compact_list). -
The class's own layout follows its fields. A compact class whose fields
are all 4 bytes or narrower is 4-aligned and rounded to 4, so
Cellis 8 bytes; one with astringor anf64keeps 8. Its objects are allocated in the range and never on the stack. -
Where a pointer is already 4 bytes, nothing moves. On wasm32
(
--profile wasi,--target wasm32), in a page module built for the host with--emit-dts(it runs on the wasm runtime, whose root trace reads each link as a pointer:compact_root_page), and underNISH_ASAN, the marker compiles to the IR the class has without it. -
The marker changes nothing under Node. Type stripping removes the
implements Compactclause, and thenish:compactimport resolves to the frozen valuesruntime/core.mjsexports, so the program runs unmodified there. -
A generic field instantiated as exactly a compact class is narrowed.
Box<Cell>.inner, withCellcompact, holds a compact link, so an interface view of the field and its implementer agree on the layout. -
A compact class is refused where C, Node or the runtime would read its
layout as full pointers, each with its own code: in a build with
--emit-header,--emit-napior--emit-napi-async, at the class (NL4021,compact_reject_header), or at an export whose signature holds one (NL4020,compact_reject_export); as adeclare class(NL2795,compact_reject_declare), or held by one (NL2796,compact_reject_declare_field); and holding aSecret, whose wipe reads it through a full pointer (NL2797,compact_reject_secret); and as a generic class, whose instantiations keep full-pointer links (NL2798,compact_reject_generic).
Wide is accepted and changes nothing yet; it is reserved for a class to
refuse a compact layout the compiler would pick on its own.
Reference: nish:compact.
Regions: nish:region
A region is memory a program frees when it says so: an arena of its own,
reached through a handle. A long-running program — a cache, a server's
per-connection state — keeps each thing that lives and dies together in one
region and gives the whole of it back with one free, where the thread's arena
would hold it until the program ends.
Reference: nish:region.
| Export | Meaning |
|---|---|
Region<T> |
a handle to a region whose root is a T: a 64-bit number, not a pointer |
region(f), regionWith(arg, f) |
a new region, whose root is what f answers; f's allocations land in it |
child(f), childWith(arg, f) |
a new region whose parent is the region entered now; called only in the f an enter or enterWith runs |
enter(r, f), enterWith(r, arg, f) |
f(root) (or f(root, arg)), allocating in the region |
free(r) |
gives the region's memory back; r and every copy of it are dead. Freeing a parent frees its children first, deepest first; freeing a child unlinks it from its parent |
f is an arrow or a named function, and an arrow captures nothing (NL2335),
so the ways into a region are its root, arg and module state. The checker
follows each (src/region-rules.ts), with two kinds of value: region memory
(the root, and whatever f allocates) and outer memory (arg, module state,
what a callee answers from outside). A number, a boolean, an enum and a
Region are copied, never pointed at, so they cross freely
(tests/cases/region_cache).
| Refused | Code |
|---|---|
region memory stored into outer memory: a field or an element, including this stored by a constructor f reaches (h.l = this). Module state cannot hold region memory at all, since a module binding is a number, boolean or string constant or module storage, whose elements are pointer-free, and a top-level let is refused first (reject_region_ctor_module_state) |
NL2609 (reject_region_to_outer, reject_region_ctor_this_arg, reject_region_store_string, reject_region_arrow_keep, reject_region_arrow_generic; a spawned task's region into what it shares, reject_async_spawn_store_region) |
outer memory that is not a number, boolean, enum or string literal stored into region memory: a field or element store, a constructor argument (new Game(g.host)) or a field of a literal |
NL2610 (reject_region_outer_string_ctor, reject_region_outer_string_literal, a spawned task's shared object kept by its own memory, reject_async_spawn_keep_holder, reject_region_arrow_link, reject_region_arrow_result, reject_region_arrow_compose, reject_region_arrow_twice, reject_region_fixpoint_chain) |
enter's or enterWith's f answering anything but a number, a boolean, an enum or a Region: an array, an object or a string |
NL2611 (reject_region_enter_result, reject_region_enter_string) |
scope(), spawn, a parallel body or a channel inside f, directly or through a callee |
NL2612 (reject_region_threads) |
a value whose type holds a Region at any depth crossing to another thread: a task's argument or answer, a parallel body's capture, a value sent on a channel |
NL2613 (reject_region_task_arg_field, reject_region_task_answer, reject_region_channel_map, reject_region_parallel_capture) |
region's or regionWith's f answering anything but memory it allocated: region(() => moduleValue), regionWith(g, (a) => a) |
NL2614 (reject_region_root_module, reject_region_root_arg) |
outer storage growing inside f, directly or through a callee: push on an outer array (arrays have no unshift or splice; the rule names them so it holds if they come), Map.set or Set.add on an outer Map or Set |
NL2615 (reject_region_grow_push, reject_region_grow_map, reject_region_grow_set, each with a _callee twin, reject_region_arrow_grow; a spawned task growing what it shares, reject_async_spawn_grow) |
a call inside f the region rules have no facts for. No program reaches it today, since every function, instantiation, lifted arrow and declare function has whole-program facts; it stays so that such a callee is never let through unexamined (tests/wordings/unreachable.txt) |
NL2616 |
Arena.mark or Arena.used inside f, or a callee reaching one: inside a region they would read the region's own arena, not the thread's |
NL2617 (reject_region_arena_mark through region, reject_region_arena_used_callee through enter and a callee) |
new Region<T>() |
NL2618 (reject_region_new) |
child or childWith outside the f an enter or enterWith runs, or in a function that something other than an enter also calls: with no region entered there is no parent |
NL2702 (reject_region_child_outside, reject_region_child_callee; a helper making the child, called from main, reject_region_child_helper_main, or from the function enter runs, reject_region_child_helper_f) |
region, regionWith, child, childWith, enter, enterWith or free in a parallel body or a spawned function, directly or through a callee |
NL2351 (reject_region_enter_spawned, reject_region_parallel_callee, reject_region_child_parallel, reject_region_child_spawned, reject_region_child_with_parallel, reject_region_child_plain_spawned) |
using r = region(f), in any declarator: a block does not free a region |
NL2387 (reject_region_using_first, reject_region_using_second) |
-
Arena.markandArena.usedare refused insidef(NL2617). Inside a region they read the region's arena, not the thread's, so a mark taken there names nothing the thread's arena knows oncefreturns (docs/security/runtime.md, RT-45). -
An arrow handed to a function runs inside
f, and is judged asf's own code. Its body is walked with what it captures, its parameters take what the call is handed, what any arrow of the same call answers (the callee may feed one arrow's answer to another, or to itself) or memory the callee allocated, and what it answers is part of what the call answers, sopick(() => a.node)stored in the root is NL2610 asa.nodeis (reject_region_arrow_keep,_grow,_link,_result,_generic,_compose,_compose_temp,_twice,_twice_temp). The walk runs to a fixpoint however long a chain of assignments is, never a fixed number of rounds (reject_region_fixpoint_chain). -
The rules are on types for threads, on values inside
f. NL2613 follows the type, so a handle hidden in a field of a task's argument is caught where it crosses, whatever route brought it there. A handle stored in region memory, or answered byenter, stays on its thread and is allowed. -
using a = arena()insidefis allowed, and so isenterof a second region: only numbers come out of it. -
A child region may point into its parent's memory that existed before the
current
enter: what the rootenterhanded in reaches, and, in the child'sf, whatchildWith'sargreaches when the root reaches it (tests/cases/region_child_keep,region_child_chain,region_child_map). The parent outlives the child, since freeing the parent frees the child, and nothing can release that memory while the parent lives: a scope opened during anenterrewinds only to a mark taken after theenterbegan, so it releases only what thisenterallocated. That memory new in thisenteris what the child may not keep (NL2610,reject_region_child_new_memory). The other rules hold the parent as outer memory: the child's memory stored into the parent's is NL2609 (reject_region_child_to_parent), growing a parent array,MaporSetin the child'sfis NL2615 (reject_region_child_grow), and the parent's memory as the child's root is NL2614 (reject_region_child_root_parent, andreject_region_child_root_listed, where it is read back out of a list the child'sfmade). The child'sfmay keep the parent's memory in the root it allocated, by a field or element store or apushonto that root's own array (tests/cases/region_child_builder_list). Onlyf's own body may: a function it calls that keeps what it is handed in the memory it answers makes that answer NL2614, underregionWithas underchildWith(reject_region_helper_keeps_arg,reject_region_child_helper_keeps_arg). A plainregion()made insideenterstays independent of the region entered, and keeps the rules above. -
What the root reaches counts as older than this
enteronly whilefstores nothing into memory the root may reach. Oncefstores into memory that may be older than thisenter— a field,push, or a callee that writes into what it is handed, such asMap.set— anywhere in its body, a value read off the root may be one it allocated, sochildWithhanding it is NL2610 (reject_region_child_root_readback,reject_region_child_root_push). A store into memoryfmade counts only whenfalso puts the root's reach (other than a string) into memory it made, as[root]ornew Holder(root)does (tests/cases/region_child_fresh_list,tests/cases/region_child_local_map); then a store through it may put new memory in the root's reach (reject_region_child_alias_field,reject_region_child_alias_element). The check is not ordered: a store after thechildWithcounts too, and one such store makes everychildWithinfhand outer memory rather than the parent's. To collect root nodes in a list, collect them in the child'sf, as above. A call's result handed tochildWithis memory thisentermay have allocated, even when the callee only reads the root, so a child may not keep it. -
In an
enterof a handle that may be a child's, the root's reach may be its ancestors' memory as well as the child's own. A child made there may keep it, but it may not grow (NL2615), keep memory other than an ancestor's (NL2610 for outer memory,reject_region_child_handle_in; NL2609 for memory new in thisenter,reject_region_child_handle_new), nor be stored into outer memory (NL2609,reject_region_child_handle_out). Nor may such a root grow: a handle that reachesenterthrough a parameter is one, even when every caller hands a plain region (NL2615,reject_region_child_handle_param). A handle may be a child's when its type is onechildorchildWithanswers somewhere in the program, unless the handle is a call ofregionorregionWithor a variable bound only ever to one: a parameter, a field or an element of that type may hold a child's (tests/cases/region_child_plain). -
Every call checks its handle at run time.
enter,enterWithandfreestop the program on a handle that is 0, freed already, or another thread's;enterof a region already entered andfreeof an entered one (or of a parent with an entered child) stop it too, so freeing twice is a panic and never a double free. A child's handle is dead once its parent is freed (tests/link/region_child_stale), andchildwith no region entered stops the program. Each call is aregionsite under--deny-panics(reject_deny_panics_region), and no proof removes it. Past 16,777,216 live regionsregionstops the program as well. -
A program run under Node as it is written (
runtime/nish.mjs) gets a region as an entry in a table,freedrops it, and the same handle checks throw.--emit-tsoutput is different: the runtime it imports (nish/runtime) has no regions yet, so it refuses the module (NL4015,reject_emit_ts_region).
process
The reference for each member of process, with its parameters, its answer and its capability, is its doc comment in runtime/nish.d.ts; the rules are here.
| Member | Takes | Gives | Effect |
|---|---|---|---|
process.exit |
i32 |
void |
write, noreturn |
process.argv |
(property) | string[] |
read |
process.platform |
(property) | string |
none |
process.arch |
(property) | string |
none |
process.exit(code: i32): void ends the program with code. The stdio
buffers of linked C code are flushed; the arena is abandoned. It is used in
statement position, and it is a terminator.
- Lowering:
nish_exit-> libcexit(code). - Tests:
process_exit,reject_exit_unreachable.
process.argv: string[] is the command line. process.argv[0] is the
program path (C's argv[0], one index earlier than Node, whose argv[0] is
the node binary and argv[1] the script), and the rest are the arguments
as UTF-8 byte strings. It is read-only.
-
Tests:
argv_echo(run withargv_echo.argv),link/argv_import;reject_argv_assign,reject_argv_push(`process.argv` is read-only).
process.platform: string is the operating system the program is
running on, spelled as Node spells it: "linux", "darwin", or
"unknown" for anything this compiler has no target triple for (every WASI
build included).
-
Tests:
io_host;reject_platform_assign(Cannot assign to `process.platform`).
process.arch: string is the architecture, likewise: "x64",
"arm64", or "unknown".
-
Tests:
io_host;reject_arch_call(Unknown builtin `process.arch`).
process.platform and process.arch are what --target host asks the
machine (WP14 §7a): the pair maps to x86_64/aarch64 plus
-unknown-linux-gnu/-apple-darwin, and anything else means this compiler has
no triple for the host and says so. Each is one call to the runtime, which
answers the address of a string in its own constant data — settled when
runtime.c was compiled, so a cross build reports the target, and nothing is
allocated or loaded, which is why the declarations are readnone and a
function built only from these stays pure. They carry no rule about the entry
point, unlike process.argv below: there is nothing for a wrapper to build, so
a wasm or N-API library may read them too. Neither can be assigned to.
process.argv is an ordinary string[] value (length, a[i], for...of,
passing it to functions, aliasing it with const args = process.argv) that
the @main wrapper builds once with nish_argv_init(argc, argv) before the
program runs; every module of the program may read it, and each read is one
load of the runtime global @nish_argv (a memory read: the function is at
most readonly). The array lives outside the arena, so no arena scope's
release invalidates it. Storing into it (process.argv[i] = s, op=,
++/--, push) is rejected. A program without export const main, as
for a wasm or N-API library, has nothing to build it from and rejects every
use with `process.argv` requires a `main` entry point (reject_argv_no_main,
tests/link/argv_no_main). Under the wasi build profile the WASI host
supplies the arguments (examples/wasi-host.mjs).
Reference: nish:process.
File I/O
Reference: nish:fs.
| Function | Takes | Gives | Effect |
|---|---|---|---|
readFileSync |
string |
string |
write |
readFileSyncOrNull |
string |
string | null |
write |
readFileBytesSync |
string |
u8[] | null |
write |
writeFileSync |
string, string |
void |
write |
appendFileSync |
string, string |
void |
write |
writeFileBytesSync |
string, readonly u8[] |
i32 |
write |
renameSync |
string, string |
i32 |
write |
unlinkSync |
string |
i32 |
write |
writeFileSyncMode |
string, string, i32 |
i32 |
write |
createFileExclusiveSync |
string, string, i32 |
i32 |
write |
mkdirSyncMode |
string, i32 |
i32 |
write |
fileOpen |
string |
i32 |
write |
fileReadAt |
i32, u8[], i32, i32, i64 |
i32 |
write |
fileSize |
i32 |
i64 |
write |
fileClose |
i32 |
i32 |
write |
readFileSync(path: string): string reads the whole file as one arena
string. A failure prints nish: cannot read <path> to stderr and exits 1.
-
Tests:
io_files;reject_readfile_number(`readFileSync` expects an argument of type string, got i32).
readFileSyncOrNull(path: string): string | null is the same read,
with null where the other exits (WP14 B3): null for exactly the set of
paths readFileSync prints nish: cannot read <path> for, a directory
included. The result is narrowed with if (text !== null) like any other
nullable.
- Tests:
io_streams;reject_readfile_or_null_unchecked.
readFileBytesSync(path: string): u8[] | null answers the file's
bytes, as they are on disk, or null for exactly the paths
readFileSyncOrNull answers null for; there is no exiting twin. The
result is an ordinary arena u8[] with length === capacity, narrowed with
if (b !== null). A wasm32 build has no readFileBytesSync, as it has no
readFileSync. The TypeScript reading is class A (WP34 N2).
-
Tests:
bytes_read,bytes_read_import,mem_read_bytes_scope;reject_bytes_read_unchecked,reject_bytes_read_arity(NL2061),reject_bytes_read_type(NL2268).
writeFileSync(path: string, data: string): void creates or truncates
the file (0644) and writes; it is used in statement position. A symbolic
link as the last component of path is refused
(nish: cannot write <path>), where Node follows it.
- Tests:
io_files.
appendFileSync(path: string, data: string): void creates or appends
and writes; it is used in statement position. A symbolic link as the last
component is refused, as for writeFileSync.
- Tests:
io_files.
writeFileBytesSync(path: string, bytes: readonly u8[]): i32 creates or
truncates the file (0644) and writes the array's bytes as they are. A
symbolic link as the last component is refused (-40 ELOOP). Where
writeFileSync exits, this answers bytes.length or a negative errno,
never a panic.
-
Tests:
io_write_file_bytes,io_write_bytes;reject_write_file_bytes_arity,reject_write_file_bytes_type(NL2268),reject_write_file_bytes_wasm.
renameSync(from: string, to: string): i32 is rename(2): from
replaces to in one step, so a reader sees the old file or the new one and
never neither. It answers 0, or a negative errno; -18 EXDEV across
file systems, since nothing is copied. Never a panic.
-
Tests:
io_rename,caps_fs_replace;reject_rename_arity,reject_rename_type,reject_rename_wasm.
unlinkSync(path: string): i32 is unlink(2): it removes the name, and
a symbolic link itself rather than what it names. A directory is refused
(-21 EISDIR on Linux, -1 EPERM where the kernel answers so), with no
fallback to rmdir. It answers 0, or a negative errno. Never a panic.
-
Tests:
io_unlink,caps_fs_replace;reject_unlink_arity,reject_unlink_type,reject_unlink_wasm.
writeFileSyncMode(path: string, data: string, mode: i32): i32 is
writeFileSync with the file left at exactly mode. An fchmod(2) sets it
before the first byte is written, so the umask cannot widen it and a
wider existing file is narrowed before it holds the new contents. A
descriptor opened before is not revoked. The file is written in place: a
failure leaves it empty or partial, and a file this process may write but
does not own is emptied before -1 EPERM, so a secret that replaces another
goes through a temporary and renameSync. A symbolic link as the last
component is refused (-40 ELOOP), its target untouched. A mode outside
0..0o777 is -22 (the setuid, setgid and sticky bits are refused, not
applied). It answers the length written, or a negative errno. Never a panic.
-
Tests:
io_write_file_mode,caps_fs_mode,fs_mode(modes read back bystat, umask022and077);reject_write_file_mode_arity,reject_write_file_mode_type,reject_write_file_mode_wasm.
createFileExclusiveSync(path: string, data: string, mode: i32): i32 is
O_CREAT|O_EXCL at exactly mode, set as for writeFileSyncMode. It
answers -17 EEXIST when any name is already there, a dangling symbolic
link included, which is never followed. Exactly one of any number of racing
processes wins, so it is a lock file; a winner whose write fails removes the
file before it answers. It answers the length written, or a negative errno.
Never a panic.
-
Tests:
io_create_exclusive,fs_mode(eight children race, one wins; a failed write leaves no file);reject_create_exclusive_arity,reject_create_exclusive_type,reject_create_exclusive_mode_type,reject_create_exclusive_wasm.
mkdirSyncMode(path: string, mode: i32): i32 is mkdir(2) of one level
at mode, with the process umask applied. An existing name is -17 EEXIST,
its mode never changed; a mode outside 0..0o777 (the sticky, setuid and
setgid bits included) is -22. It answers 0, or a negative errno. Never a
panic.
-
Tests:
io_mkdir_mode,caps_fs_mode,fs_mode;reject_mkdir_mode_arity,reject_mkdir_mode_type,reject_mkdir_mode_mode_type,reject_mkdir_mode_wasm.
A file a window at a time
readFileBytesSync answers the whole file; these four read it a window at
a time into a u8[] the program owns, so memory follows the window rather
than the file, and a file past 2^31 − 1 bytes is readable (WP44,
wp44-streaming-files.md). No call allocates. Every
answer that is not a count is the negated Linux errno, on every platform,
WASI and Node included: -2 not found, -6 a Unix-domain socket (ENXIO),
-9 a bad descriptor, -13 permission denied, -20 a component is not a
directory, -21 a directory, -22 not a regular file or a refused position,
-23/-24 descriptors exhausted, -29 not seekable, -36 name too long,
-40 too many symbolic links, and -5 for any other failure, WASI's
ENOTCAPABLE among them. They are exported by nish:fs and global.
fileOpen(path: string): i32 opens a regular file for reading and
answers its descriptor (>= 0), or a negative errno. It opens with
O_CLOEXEC, so a spawned child does not inherit it, and O_NONBLOCK, so a
FIFO with no writer cannot hang it. Anything that is not a regular file is
closed again and refused: a directory is -21, a FIFO or a device -22, and
a Unix-domain socket -6, as open(2) answers for one. An empty path, or one
holding a NUL, is -2 before the file system is asked. Symbolic links are
followed.
- Tests:
file_read_at,file_open_errors.
fileReadAt(fd: i32, buf: u8[], off: i32, len: i32, pos: i64): i32 reads
at most len bytes of the file from byte pos into buf[off, off + len),
with pread(2): there is no cursor, so one descriptor serves any number of
readers. It answers the count, 0 at or past the end of the file, or a
negative errno. A short read is allowed before the end: loop until 0, or
until the size planned for is read. len of 0 answers 0. A pos below
0, or a pos + len past 2^63 − 1, is -22 before the kernel is asked. The
window is checked as netRead's is, and buf is written, so a
readonly u8[] is refused (Cannot `fileReadAt` through readonly u8[])
and another element type is NL2402, as netRead's are.
-
Tests:
file_read_at,file_read_at_eof,file_read_at_pos,file_read_at_bounds,file_size_large,mem_file_stream_flat;reject_file_read_readonly,reject_file_read_element.
fileSize(fd: i32): i64 answers the file's size in bytes now, or a
negative errno. The size can change: a file may shrink or grow between
fileSize and the reads, so a reader that gets 0 before the size it planned
for has a file that shrank, and a server that has sent Content-Length must
close the connection there.
- Tests:
file_read_at,file_size_large.
fileClose(fd: i32): i32 closes the descriptor and answers 0 or a
negative errno; either way the descriptor is gone. It is never retried. A
call on a closed descriptor is -9, as is one on a negative descriptor
(under Node too, where fs would throw a range error), but nothing checks a use after close: the
number may already name another file the program opened since (recorded
open, docs/security/runtime.md, RT-50).
- Tests:
file_read_at_pos.
All four are fs.read (capabilities; caps_file_read,
reject_file_read_denied) and write effects, so none may be called in a
scope() task or a parallel body (reject_file_fd_type for the descriptor's
i32, reject_file_read_task). Under Node they read through fs.openSync,
fs.readSync, fs.fstatSync and fs.closeSync, with the size a bigint.
Under --profile wasi they read through WASI's path_open, fd_pread and
fd_filestat_get, from the directories the host preopens. A freestanding
wasm32 build has no file system to read, and a call is refused at check
time: `fileOpen` reads a file, and a freestanding wasm32 build has no file system to read (build with `--profile wasi`) (NL2718;
reject_file_open_wasm). The --profile wasm runtime defines the four
answering -38 (ENOSYS), so a link that names them never fails.
The ones that answer an i32 answer the platform's errno values,
untranslated, and the same numbers under Node; the ones shown here are
Linux's, so compare against a negative result. A wasm32 build has none of
these, and a WASI build answers -ENOSYS from the three that take a mode, as
WASI has no file modes to set.
Bytes out, and the string route. A native string is {len, bytes},
never validated as UTF-8, and write, writeError, writeFileSync and
appendFileSync send its len bytes as they are, so a string built by
String.fromBytes from any bytes, all 256 values included, leaves exactly
as it came in (tests/cases/io_write_bytes, compared with
io_write_bytes.bin byte for byte). That route is native-only. Under the
prelude (runtime/nish.mjs) String.fromBytes makes each byte one UTF-16
unit, which leaves as UTF-8, so a byte above 127 becomes two; under
--emit-ts the bytes decode as UTF-8 and a byte that is not UTF-8 becomes
U+FFFD. writeBytes and writeFileBytesSync send a u8[] as it is on
every host, and are the route for bytes that are not text.
Paths are relative to the working directory. readFileSync,
readFileSyncOrNull and readFileBytesSync treat a file longer than
2^31 − 1 bytes as one they cannot read, in both number modes, which matches
Node. Every path, name and argument a builtin hands the OS that holds a NUL
byte is refused as a missing path is — null, false, -1 or
cannot …, whichever that builtin answers — where Node refuses it by
throwing (docs/security/runtime.md, RT-1, RT-3, RT-4). These are globals, not
import { readFileSync } from "fs" (bare imports are rejected), and so are
write, writeError and panic above. A user function of the same name
wins, as it does for every identifier builtin.
Directories and subprocesses
Reference: nish:fs, nish:process.
| Function | Takes | Gives | Effect |
|---|---|---|---|
mkdirSync |
string |
boolean |
write |
isDirectorySync |
string |
boolean |
write |
spawnSync |
string[] |
number |
write |
spawnSyncTo |
string[], string, string |
number |
write |
spawnPiped |
string[], readonly u8[], u8[], u8[] |
i32 |
write |
readdirSync |
string |
string[] | null |
write |
realpathSync |
string |
string | null |
write |
mkdirSync(path: string): boolean creates one directory, mode
0777 & ~umask, not recursive, exactly like Node's fs.mkdirSync(p) with
no options. It answers true when a directory exists at path once the
call returns, whoever made it; false for every other outcome, a plain file
at path included.
- Tests:
io_mkdir;reject_mkdir_arity,reject_mkdir_type.
isDirectorySync(path: string): boolean does one stat: true when a
directory exists at path as the call runs, false for everything else. It
is the question -o <dir> asks of a path, and the stat half of
mkdirSync, which calls it (WP14 §7a).
-
Tests:
io_is_directory;reject_is_directory_arity,reject_is_directory_type.
spawnSync(argv: string[]): number runs argv[0], searched on PATH,
and waits for it. It answers its exit status, or 128 + n when signal n
killed it (the shell's convention). It answers -1 for an empty vector,
a child that cannot be started or waited for, and always on WASI, which has
no processes. The child inherits this process's environment, streams and
working directory.
- Tests:
io_spawn;reject_spawn_arity,reject_spawn_element_type.
spawnSyncTo(argv: string[], stdoutPath: string, stderrPath: string): number
is the same run, with each stream sent to a file created or truncated at
0644. An empty path leaves that stream inherited, and a
symbolic link as a path's last component is refused, which answers -1.
The two paths must not name one file; capture them apart and concatenate.
-
Tests:
io_spawn_to,io_spawn_to_inherit;reject_spawn_to_arity,reject_spawn_to_type.
spawnPiped(argv: string[], stdin: readonly u8[], stdout: u8[], stderr: u8[]): i32
is the same run, with every stream a pipe: the child reads
stdin and then end of file, and its stdout and stderr are appended to
stdout and stderr. It answers the status, 128 + n, or -1 as for
spawnSync (and for a failed pipe or poll). A stream past 16 MiB
(NISH_PIPED_CAP) kills the child with SIGKILL and answers -2. The
child holds no descriptor of the program but its three streams, and the
program's SIGPIPE from a child that stopped reading is absorbed. The two
output arrays are grown by the runtime, so they are push receivers to the
arena analysis, and readonly is refused for them, as is a typed-array
spelling (Int32Array, ...), whose length is fixed.
-
Tests:
io_spawn_piped,mem_spawn_piped_scope;reject_spawn_piped_arity,reject_spawn_piped_element_type,reject_spawn_piped_stdin_type,reject_spawn_piped_output,reject_spawn_piped_typed_output,reject_spawn_piped_wasm,reject_using_arena_spawn_piped.
spawnPipedWithin(argv: string[], stdin: readonly u8[], stdout: u8[], stderr: u8[], timeoutMs: i32): i32
is spawnPiped's run with a deadline timeoutMs milliseconds after the
call, on the monotonic clock; a negative timeoutMs is none, and the run
is spawnPiped's. It gives every answer spawnPiped gives, and -3 when
the deadline passes first: the child's pid is killed with SIGKILL and
reaped, and the bytes read before then stay in stdout and stderr. The
deadline covers the whole run, the wait for the child after its streams
close included, so a child that closes its output and keeps running is -3
too. Only the child's own pid is killed, never a process group: a grandchild
still holding an output pipe keeps the run going until the deadline, which
turns an exit 0 into -3, and is left running. The arguments are checked as
spawnPiped's, the fifth as an i32.
-
Tests:
io_spawn_piped_within;reject_spawn_piped_within_arity,reject_spawn_piped_within_element_type,reject_spawn_piped_within_timeout_type,reject_spawn_piped_within_output,reject_spawn_piped_within_typed_output,reject_spawn_piped_within_wasm,reject_using_arena_spawn_piped_within.
readdirSync(path: string): string[] | null answers the directory's
entries, sorted ascending by bytes, without . and ... It answers
null when the directory cannot be read at all, a different answer from
the empty array of an empty directory. A WASI build always answers null:
fd_readdir lists a preopened directory rather than a path, which is a
different contract and not a port of this one.
-
Tests:
io_readdir,io_readdir_null;reject_readdir_arity,reject_readdir_type,reject_readdir_unchecked.
realpathSync(path: string): string | null answers path resolved to
an absolute path with no symbolic link, ., .. or repeated
separator, or null when it does not resolve. POSIX realpath needs every
component but the last to exist, so "does not resolve" is one answer rather
than several, and the answer the caller asked for rather than an error. The
result is copied into the arena and narrowed with if (p !== null) like any
other nullable (WP19 §5a item 4).
- Tests:
io_realpath;reject_realpath_arity,reject_realpath_type.
All five answer a value where they could have exited, for the reason
readFileSyncOrNull answers null (WP14 B3): the language has no exceptions,
so a driver has to be able to turn the failure into its own diagnostic. They
exist so that a compiler written in Nish can create its own -o dir/,
tell -o out meaning a directory from -o out meaning a file, and shell out
for --link — mkdirSync and spawnSync are the calls
wp14-selfhost.md §3a D4 named as the price of a
self-hosted link step, and isDirectorySync is the one §7a named as the price
of the last spelling of -o that, at the time, only the TypeScript compiler
handled.
spawnSync and spawnSyncTo are the two builtins whose pointer argument the
runtime keeps: the spawn path copies each element's bytes pointer into a vector
that outlives the call, so the array escapes, a parameter passed to either is
never nocapture, and an array literal handed to one is never stack-allocated
(io_spawn pins all three). Nothing that can reach either is willreturn
either, because the child may never exit. The argument is checked down to its
element type: an i32[] is an array but not a command line
(reject_spawn_element_type). Arguments are passed as bytes, so an embedded NUL
truncates one.
spawnSyncTo is a second builtin rather than two more parameters on the first
because the language had no default parameters when it was added — and the alternative shape, answering
the output as a string, was refused for two reasons: the runtime would have to
buffer a child that can print more than memory holds, and a harness comparing
output against a golden wants the bytes on disk anyway, to diff and to keep
after a failure.
spawnPiped is the run whose streams stay in memory, for a child that takes
a secret: a password manager's or keychain's CLI reads it on stdin, and stdin
is the one place ps, /proc/<pid>/environ and the file system do not show.
Its shape is four arguments with the outputs as out-parameters, the way
readBytesInto fills a caller's array. The rejected shape answered a record
({ status, stdout, stderr }): no builtin returns a class, and there is no
way to declare one the checker, both emitters and the TypeScript prelude would
share, so the record would have been the first and its own construct. The
out-parameters cost a caller two const out: u8[] = [] lines; they also let a
loop reuse one pair of arrays. The runtime writes stdin and drains both
outputs in one poll loop, so a child that fills one pipe while the parent
waits on another cannot deadlock it, and the 16 MiB cap is the answer to the
reason spawnSyncTo gave for refusing to buffer: a child that prints more
than memory holds is stopped, with a code no exit status can be, rather than
followed (docs/security/runtime.md, RT-27 to RT-31).
spawnPipedWithin is the same run for a child that may never answer: a
credential helper waiting on a prompt nobody sees, or a tool hung on the
network. It is a fifth argument on a second builtin rather than an optional
one on spawnPiped, so spawnPiped's IR and every program that calls it are
unchanged; -3 is a code of its own because a caller tells a slow child from
a failed one (the deadline was the program's choice, so it may retry with a
longer one). The deadline is a point in time rather than a budget per wait:
the poll loop's waits and the final wait for the child all end at it, so the
whole call returns within timeoutMs plus the time a kill and a reap take.
It kills the pid and not a process group, because putting the child in a
group of its own would take it out of the terminal's job control, and the
program cannot know the child would not mind (docs/security/runtime.md,
RT-49).
spawnWithFds numbers the descriptors from 3 rather than taking a target
for each, because 3 onward is where a child that is handed descriptors looks
for them (systemd's socket activation, LISTEN_FDS), and a target list would
let a program aim one at 0 to 2 and lose the child's streams. The runtime
duplicates each source above the targets before the child's dup2 reads it,
so an entry that is its own target or another's is never overwritten first
(docs/security/runtime.md, RT-40).
readdirSync sorts, which neither readdir(3) nor Node's fs.readdirSync
does, and that is a decision about this language rather than a convenience:
every caller of an unsorted listing would have to sort it before reading it,
and a run whose order is the file system's is a run that
differs between two machines. The sort is by bytes, like every other string
comparison here, so it agrees with Node's .sort() for ASCII names and can
differ for names that are not (docs/wp13-differential.md).
The environment
Reference: nish:process.
| Function | Takes | Gives | Effect |
|---|---|---|---|
getenv |
string |
string | null |
write |
getenv(name: string): string | null answers the value of environment
variable name as the call runs, or null when it is unset; a variable
set to nothing (CC=) answers a zero-length string. The bytes are copied
into the arena, so the result outlives any later change to the environment.
It is narrowed with if (cc !== null) like any other nullable (WP19 R1).
-
Tests:
io_getenv;reject_getenv_arity,reject_getenv_type,reject_getenv_unchecked.
It is a call and not process.env.CC, and that is a language decision
rather than a spelling: process.env.NAME is member access on a key chosen at
runtime, which Phase 0 forbids outright
(wp19-stage0-retirement.md §4). The two
process.* values the language does have — process.argv, and
process.platform / process.arch below — are fixed names with no key.
null and "" are different answers, and a driver acts on the difference: an
unset CC means "use the default", and CC= means someone set it to nothing.
That is why the result is string | null and not a string that is empty when
nothing is there.
There is no setenv: a program can read its environment and pass an
environment on to a child through spawnSync, which inherits this process's,
but it cannot change its own. Nothing in the compiler needs to, and a setter
would make getenv's "as the call runs" a much sharper caveat than it is.
Because of that, a test that depends on a variable is given one by the harness
— tests/cases/<name>.env, one NAME=value per line, read by both
tests/run.js and the differential runner so the native binary and the Node
rewrite see the same environment.
Two reads of one variable in a function are two calls: the declaration is not
readonly, because a setenv from linked C — this program cannot call one,
but a library linked beside it can — may change the answer between them, so
LLVM may not fold the second read into the first.
The value is what getenv(3) answers, so it is bytes, and every offset into
it is a byte offset.
The clock
Reference: nish:process.
| Function | Takes | Gives | Effect |
|---|---|---|---|
monotonicNanos |
nothing | i64 |
write |
wallClockNanos |
nothing | i64 |
write |
monotonicNanos(): i64 reads CLOCK_MONOTONIC in nanoseconds. The
origin is arbitrary, so only the difference between two reads means
anything.
- Tests:
io_monotonic;reject_monotonic_arity.
wallClockNanos(): i64 reads CLOCK_REALTIME in nanoseconds since
1970-01-01 UTC, as an i64. It can go backwards when the system time is
set, so a duration is monotonicNanos's job. It is not refused on wasm, as
the monotonic clock is.
- Tests:
wall_clock;reject_wall_clock_arity.
It is an i64 in both number modes and not a number, because a number is an
i32 by default and would overflow inside a tenth of a second, and an f64
loses nanoseconds after about 104 days of uptime.
It is the monotonic clock and not the wall clock, and the type cannot say so,
so the name does. An elapsed time measured from a wall clock has to survive that
clock being corrected mid-run, and it cannot: the difference can come out
negative. The wall clock is Date.now(), below, and it is the only part of
Date there is: a program that formats a date needs a calendar, a time zone
database and a locale, none of which the language has. The ISO calendar is a
library instead: nish/temporal is TC39 Temporal's dates, times, instants
and durations, with no zone but UTC yet (nish/temporal).
The effect is write rather than none for the reason getenv is not
readnone: the clock is not memory LLVM is tracking, so two reads with work
between them must not fold into one. A readnone clock would make every measured
interval exactly zero, which io_monotonic is the test for.
The host: the wall clock, entropy, file times and signals
Four facts a program that runs for days asks of the machine (WP34 N3): what time
it is, bytes nobody can predict, whether a file was renewed, and whether it has
been asked to stop. All four are C in runtime/runtime-host.c, a translation
unit of its own with its own .text* ceiling, which scripts/build.sh compiles
beside runtime.c as it does the others.
Reference: nish:fs, nish:process, and the globals page for Date.now (under DateConstructor) and crypto.
| Function | Takes | Gives | Effect |
|---|---|---|---|
Date.now |
nothing | f64 |
write |
crypto.getRandomValues |
u8[] |
void |
write |
statMtimeSync |
string |
f64 |
write |
signalFd |
nothing | i32 |
write |
readSignal |
i32 |
i32 |
write |
Date.now(): f64 answers the wall clock, CLOCK_REALTIME, in whole
milliseconds since 1970-01-01 UTC, as JavaScript answers it. The
milliseconds are counted as integers and then converted, so the value is
exact and a time before 1970 floors. It is an f64 in both number modes,
because an i32 of milliseconds overflowed in January 1970. It can go
backwards when the clock is corrected, so an interval is
monotonicNanos's job, not this one's.
- Tests:
os_now,os_now_i32;reject_os_date_now_arity.
crypto.getRandomValues(bytes: u8[]): void fills every byte of bytes
from the kernel's CSPRNG. It takes a u8[] only, where the Web API
takes any integer typed array: the byte is the unit a key or a nonce is
written in, and a wider element would make the answer's byte order a fact
about the machine. It is a statement, where the Web API also answers its
argument, which the caller already holds. An empty array is filled with
nothing. It fills at most 65,536 bytes a call, the Web API's limit: more
is a panic,
crypto.getRandomValues: <n> bytes asked for, and one call fills at most 65536,
exit 1. A source that fails is a panic too; nothing weaker is ever
substituted.
-
Lowering:
getrandom(2)on Linux, looping overEINTRand short reads, andgetentropy(3)in 256-byte chunks on macOS. -
Tests:
os_random,os_random_limit;reject_os_random_i32,reject_os_random_u32,reject_os_random_string,reject_os_random_readonly,reject_os_random_value,reject_os_random_arity,reject_os_crypto_member,reject_os_crypto_value,reject_thread_region_random.
statMtimeSync(path: string): f64 answers the modification time in
milliseconds since the epoch, with the sub-millisecond fraction kept. It
is computed as Node's mtimeMs is (seconds × 1e3 plus nanoseconds ÷ 1e6,
in doubles), so the two print the same digits. It answers NaN when the
path cannot be stat'd, because an f64 has no null
(Nullable types); test it with m !== m. stat follows a
symbolic link, and a directory has a time of its own. Also exported by
nish:fs.
-
Tests:
os_mtime,os_mtime_import;reject_os_mtime_type,reject_os_mtime_arity,reject_os_mtime_nullable.
signalFd(): i32 answers a descriptor that becomes readable when
SIGTERM or SIGINT arrives: the read end of a pipe a sigaction handler
writes one byte to per signal, so the signals' default action never runs.
The handler runs in whichever thread the signal lands on, so a scope()
task or a parallel worker running when signalFd() is called cannot take
the signal and end the process, and nothing is blocked for a spawned child
to inherit. It overrides a disposition the process started with, as
process.on("SIGTERM") does under Node. It is made once; -1 when it
cannot be made. Also exported by nish:process.
- Tests:
os_signal,os_signal_import;reject_os_signal_fd_arity.
readSignal(fd: i32): i32 blocks until one of the two signals arrives
and answers 15 or 2; -1 for an fd that is not signalFd()'s and
for a read that fails. The argument is an i32 in both number modes, and an
f64, an i64 or a string is refused rather than converted. Also exported
by nish:process.
- Lowering: not
willreturn. -
Tests:
os_signal,os_signal_import;reject_os_signal_f64,reject_os_signal_i64,reject_os_signal_string,reject_os_signal_arity.
Date has one member. new Date(), Date(), Date.parse, Date.UTC,
any other member, and Date or Date.now used as a value are all refused under
one rule: `new Date()` is refused: Nish has no `Date` object; the one `Date` member is `Date.now()`, the wall clock in milliseconds (NL2401;
reject_os_date_new, reject_os_date_call, reject_os_date_parse,
reject_os_date_utc, reject_os_date_value, reject_os_date_bare). A local
named Date shadows the global as any local does, so calling it is the
ordinary Unknown function `Date` (reject_os_date_shadowed). crypto
has one member the same way: `crypto.randomUUID` is refused: the one `crypto` member is `crypto.getRandomValues(bytes)` (NL2403). The array
getRandomValues fills is written, so a readonly u8[] is refused as a store
through it is, a parameter it fills is never readonly in the IR, and inside a
scope's region it is refused on an array a task may read, as fill is. Anything
but a u8[] is `crypto.getRandomValues` fills a `u8[]`, got i32[]
(NL2402).
A descriptor, not a handler. A handler would be a callback the runtime
calls, which a function reference is not, and would
run between any two instructions of the program. A descriptor is what a server's loop can wait on beside its sockets
(WP34 N5 will add that wait); until then readSignal is the blocking read.
Underneath it is a handler rather than a Linux signalfd, because a signalfd
hears only a signal blocked in every thread, and a mask reaches only the thread
that sets it. A child that spawnSync or spawnSyncTo starts after
signalFd() can still be stopped by either signal, because exec resets a
caught signal to its default (os_signal). Signals other than SIGTERM and
SIGINT are not in the language.
On wasm. A wasm32 target and the wasi profile have no host for these to
reach, and runtime-host.c is empty there, so all five are refused by the
checker: `Date.now` reaches the operating system, and a wasm32 build has none to reach (NL2404; reject_os_now_wasm, reject_os_random_wasm,
reject_os_mtime_wasm, reject_os_signal_fd_wasm, reject_os_read_signal_wasm).
The TypeScript reading (wp33-round-trip.md §2).
Date.now is class A: Node's is the same floored CLOCK_REALTIME, and the
os_ block of tests/run.js holds both readings between two reads of Node's
own clock. statMtimeSync is class A under runtime/nish.mjs, which answers
fs.statSync(path, { throwIfNoEntry: false })?.mtimeMs, or NaN: the os_
block sets a file's mtime to a fraction of a millisecond and requires the
native build and the prelude both to print Node's mtimeMs for it.
crypto.getRandomValues is class A under the prelude and class C without
it: Node's own takes only a typed array and a u8[] is a plain Array there,
so the prelude replaces the method on Node's crypto with one that fills a
plain array too and panics past 65,536 with the native words; os_random and
os_random_limit run unmodified under it. signalFd and readSignal are
class C, with no synchronous reading at all: Node delivers a signal only to
its event loop, which a blocking readSignal never returns to, so the prelude's
versions throw rather than answer. The row in
wp33-round-trip.md §3.5 gives the translation.
Children and the machine: spawn, kill, waitPid, availableParallelism, totalMemory
What a supervisor asks of the host (WP34 N12): start a child without waiting
for it, stop it, reap it when it ends, and size itself to the machine it runs
on. All five are C in runtime/runtime-os.c, beside spawnSync, and each is
exported by nish:process as well as being a global.
Reference: nish:process.
| Function | Takes | Gives | Effect |
|---|---|---|---|
spawn |
string[] |
i32 |
write |
spawnWithFds |
string[], readonly i32[] |
i32 |
write |
kill |
i32, i32 |
i32 |
write |
waitPid |
i32, i32 |
i32[] |
write |
availableParallelism |
nothing | i32 |
none |
totalMemory |
nothing | i64 |
none |
spawn(argv: string[]): i32 starts argv[0] exactly as spawnSync
does and answers the child's pid at once, without waiting, or a
negative errno (-22 for an empty vector or an argument holding a NUL).
Test pid > 0 before handing the answer on: a failure passed to
waitPid reads as "any child". A child never reaped stays a zombie until the
program exits.
- Tests:
os_spawn;reject_os_spawn_type,reject_os_spawn_wasm.
spawnWithFds(argv: string[], fds: readonly i32[]): i32 is spawn,
with the child holding fds[i] as descriptor 3 + i, close-on-exec
cleared there. No other descriptor of this process reaches it but one
opened without close-on-exec, which nothing the runtime opens is. More than
64 descriptors, or a negative one, is -22 before anything is opened or
started. The child shares each descriptor's open file, offset and status
flags, and this process keeps its own. It answers its pid, or a negative
errno.
-
Tests:
spawn_fds_child,caps_process_spawn_fds;reject_spawn_fds_arity,reject_spawn_fds_element,reject_spawn_fds_argv,reject_spawn_fds_wasm.
kill(pid: i32, sig: i32): i32 sends sig to pid and answers 0, or a
negative errno. It takes only 0, 2 (SIGINT), 9 (SIGKILL) and 15
(SIGTERM), whose numbers are the same on Linux and Darwin; any other sig
is -22, and so is a pid of 0 or below, which kill(2) reads as more than
one process. A positive pid may be any process of this user, not only a
child (RT-19).
- Tests:
os_kill;reject_os_kill_f64,reject_os_kill_wasm.
waitPid(pid: i32, flags: i32): i32[] reaps a child (pid a child's or
-1 for any child; flag 1 is WNOHANG) and answers a fresh
[pid, status], the status as spawnSync's; a failure puts the
negative errno first. After signalFd() a blocking wait never sees EINTR,
because that handler is installed with SA_RESTART and the wait restarts.
It is an array because the language has no tuple.
-
Tests:
os_wait,os_spawn,os_kill;reject_os_wait_arity,reject_os_wait_wasm.
availableParallelism(): i32 answers the CPUs this program may use,
at least 1: the smallest of the CPUs online, its affinity mask or
cpuset, a cgroup v2 CPU quota rounded up, and a positive
NISH_NUM_THREADS, which can lower the count and never raise it. It needs
no --threads.
- Tests:
os_cpu;reject_os_cpu_arity,reject_os_cpu_wasm.
totalMemory(): i64 answers bytes of physical memory, or -1
where the host does not say. It is the machine's, not a cgroup's limit.
-
Lowering:
_SC_PHYS_PAGEStimes the page size on Linux,hw.memsizeon Darwin. - Tests:
os_memory;reject_os_memory_wasm.
Capabilities. spawn, kill and waitPid carry process.spawn, the
label spawnSync carries: each reaches other processes — kill any process of
this user, as kill(1) does, not only a child spawn started — so none takes a
label of its own (caps_process_wait).
availableParallelism and totalMemory carry none: how many cores and
bytes the machine has is a fact about the host, as process.platform is, read
to size a program rather than to reach anything (caps_host).
In a task. spawn, kill and waitPid write state outside the program,
so a scope's task or a parallel body that calls one — under any import alias —
is refused as any shared write is (reject_os_spawn_task,
reject_os_kill_task, reject_os_wait_task, reject_os_kill_task_alias,
reject_os_kill_parallel). availableParallelism and totalMemory write
nothing and may be called there (tests/link/thread_host_facts_task).
On wasm. A wasm32 target has no processes and no host to ask, so all five
are refused by the checker with the N3 rule's words: `spawn` reaches the operating system, and a wasm32 build has none to reach (NL2404;
reject_os_spawn_wasm, reject_os_kill_wasm, reject_os_wait_wasm,
reject_os_cpu_wasm, reject_os_memory_wasm).
The TypeScript reading. kill, availableParallelism and totalMemory
are class A under runtime/nish.mjs: kill checks the signal and the pid
as the native one does and turns Node's error code back into its errno;
availableParallelism is os.availableParallelism() lowered by
NISH_NUM_THREADS (Node reads no cgroup quota, so inside a container with one
the two can differ); and totalMemory is os.totalmem(), a bigint as every
i64 is there. spawn and waitPid are class C: Node reaps its children
itself, on the event loop, so a synchronous wait has nothing to find, and the
prelude's versions throw, naming themselves, rather than answer (the os_
block of tests/run.js).
Scheduling: setNice, setIdlePriority, setCpuAffinity
How much of the CPU the program asks for, and where: a batch job or a background indexer
steps aside for the interactive work on the machine, and a prefork worker keeps to one core. Each is C in
runtime/runtime-os.c, and each is exported by nish:process as well as
being a global.
Reference: nish:process.
| Function | Takes | Gives | Effect |
|---|---|---|---|
setNice |
i32 |
i32 |
write |
setIdlePriority |
nothing | i32 |
write |
setCpuAffinity |
i32 |
i32 |
write |
setNice(n: i32): i32 sets the nice value with
setpriority(PRIO_PROCESS, 0, n) and answers 0, or a negative errno. It is
lower-only, even as root: outside 0..19 is -22 and below the current
value is -13, both with no system call, so the priority never rises.
Children started afterwards inherit it.
- Tests:
os_nice;reject_os_nice_wasm,reject_os_nice_task.
setIdlePriority(): i32 moves the program to SCHED_IDLE with
sched_setscheduler(0, SCHED_IDLE, …) and answers 0 or a negative errno. It
is Linux only: Darwin answers -95 (Linux's EOPNOTSUPP, as nish:net
spells it). An unprivileged process can never leave SCHED_IDLE, and an
idle process can starve on a saturated machine, so a caller keeps a
normal-priority fallback for work that must finish.
- Tests:
os_nice;reject_os_idle_wasm.
setCpuAffinity(cpu: i32): i32 pins the calling process to the one CPU
cpu with sched_setaffinity and answers 0, or a negative errno. It is
narrow-only: a cpu outside the CPUs the process may already run on
(taskset, a cgroup) is -22 with no change. availableParallelism()
answers 1 afterwards. It is Linux only: -95 on Darwin and under Node.
-
Tests:
link/net_prefork_pin;reject_os_affinity_wasm,reject_os_affinity_task.
Per thread. Linux keeps both the nice value and the policy per thread: a
call sets the calling thread's and every thread started after it. So both
belong at the top of main, before a scope() or a parallel loop starts the
pool, and a task or a parallel body that calls one is refused as any shared
write is (reject_os_nice_task). Neither ever crashes the program: a failure
is only the answer.
Capabilities. Both carry process.spawn, the label kill carries: neither
reaches another process, but the priority is the one the program's children
inherit, and a step down cannot always be taken back.
On wasm. A wasm32 target schedules nothing of the host's, so both are
refused by the checker with the N3 rule's words (NL2404;
reject_os_nice_wasm, reject_os_idle_wasm).
The TypeScript reading. Both are class C under runtime/nish.mjs:
Node has os.setPriority but no SCHED_IDLE, and a reading of one without
the other would differ from the native build on the same host, so both throw,
naming themselves.
nish:net: addresses, non-blocking TCP and UDP, and the readiness loop
Sockets for a program that owns its loop (WP34 N5): a server listens, accepts
and reads and writes bytes, a client connects, or either binds and trades
datagrams, and every socket is
non-blocking, so a call that
would wait answers "would block" instead, and the program decides what to do
next. There is no callback, and an async function
cannot wait on a socket yet (its tasks run on an in-process queue): the program
is the loop, and it waits in pollWait for
whichever of its descriptors is ready. Every socket is also close-on-exec,
so a child that spawnSync starts inherits none of them. All twenty-six functions
are C in
runtime/runtime-net.c, a translation unit of its own with its own .text*
ceiling, which scripts/build.sh compiles beside runtime.c as it does the
others; section GC leaves every one of them out of a program that calls none
(the net_ block of tests/run.js links one and counts the symbols). Each is
exported by nish:net and is a global too, as every nish: export is.
Reference: nish:net.
| Function | Takes | Gives | Effect |
|---|---|---|---|
netAddress |
u8[], string, i32 |
i32 |
write |
netLocalPort |
i32 |
i32 |
write |
tcpListen |
string, i32, i32 |
i32 |
write |
tcpListenFlags |
string, i32, i32, i32 |
i32 |
write |
tcpAccept |
i32, u8[] |
i32 |
write |
netRead |
i32, u8[], i32, i32 |
i32 |
write |
netWrite |
i32, readonly u8[], i32, i32 |
i32 |
write |
netWrite2 |
i32, two ranges |
i32 |
write |
netShutdown |
i32, i32 |
i32 |
write |
netClose |
i32 |
i32 |
write |
netSetOption |
i32, i32, i32 |
i32 |
write |
netMss |
i32 |
i32 |
write |
unixListen |
string, i32, i32 |
i32 |
write |
unixListenMode |
string, i32, i32, i32 |
i32 |
write |
unixConnect |
string |
i32 |
write |
unixAccept |
i32 |
i32 |
write |
unixSocketPair |
i32, i32[] |
i32 |
write |
sendFd |
i32, i32, readonly u8[], i32, i32 |
i32 |
write |
recvFd |
i32, u8[], i32, i32, i32[] |
i32 |
write |
peerCredentials |
i32, i32[] |
i32 |
write |
netRingState |
none | i32 |
write |
netAddress(out: u8[], host: string, port: i32): i32 writes the
address form of a numeric host and port into
out. It answers 0, or -22, leaving out as it was. No nish:net call
looks a name up; nish/net/dns resolves it first.
- Tests:
net_tcp_calls,net_tcp_import;reject_net_unknown_export.
netLocalPort(fd: i32): i32 answers the port the socket is bound to: the
one the kernel chose for a listen on port 0.
- Tests:
net_tcp_calls,net_tcp_echo.
tcpListen(host: string, port: i32, backlog: i32): i32 answers a
listening socket on a numeric host with SO_REUSEADDR. "::" hears both
families, and on a kernel without IPv6 it is IPv4's 0.0.0.0. It answers
-98 when the port is in use.
-
Tests:
net_tcp_calls,net_tcp_echo,net_dual_stack;reject_net_arity,reject_net_wasm.
tcpListenFlags(host: string, port: i32, backlog: i32, flags: i32): i32
answers tcpListen's socket, with flags. 1 is SO_REUSEPORT, which
differs by platform as the table below shows. 2 is TCP_NODELAY, which the
runtime sets on every accepted connection, on every platform, closing one on
which it cannot. 4, with 1, steers each connection to the listener of the
CPU that received it, Linux only (-95 elsewhere, -22 without 1). 8
(NET_LISTEN_RING) accepts into io_uring slots on Linux, as
slot handles say, and does nothing elsewhere. Any other bit
is -22 before a socket is made.
-
Tests:
net_tcp_reuseport,net_tcp_nodelay,net_ring_calls,link/net_tcp_reuseport_spread,link/net_tcp_nodelay_shared;reject_net_reuseport_wasm,reject_net_nodelay_wasm.
tcpAccept(fd: i32, peer: u8[]): i32 answers the next waiting
connection, non-blocking and close-on-exec, with the peer's address in
peer. It answers -11 when none is waiting, and a peer shorter than 18
bytes is -22 before a connection is taken.
-
Lowering: not
willreturn: on a blocking socket the program inherited it waits. - Tests:
net_tcp_calls,net_tcp_echo;reject_net_accept_readonly.
netRead(fd: i32, buf: u8[], off: i32, len: i32): i32 reads at most
len bytes into buf[off, off + len). It answers the count, 0 at the end
of the stream, or -11 when nothing is waiting. The range is
checked.
-
Tests:
net_tcp_echo,net_tcp_bounds;reject_net_readonly,reject_net_element,reject_net_import_wasm.
netWrite(fd: i32, buf: readonly u8[], off: i32, len: i32): i32 writes
at most len bytes of buf[off, off + len). It answers the count, which may
be short. A peer that has gone is -32, never SIGPIPE, so a closed
connection cannot end the process. The range is checked.
- Tests:
net_tcp_echo,net_tcp_bounds;reject_net_write_element.
netWrite2(fd: i32, a: readonly u8[], aOff: i32, aLen: i32, b: readonly u8[], bOff: i32, bLen: i32): i32
writes two ranges in one system call, answered as netWrite is, -32
and never SIGPIPE included. Both ranges are checked, a's first.
As for every call here, a wasm32 build refuses it and Node throws.
- Lowering:
sendmsgwith twoiovecs. -
Tests:
tests/link/net_write2,net_write2_bounds;reject_net_write2_element,reject_net_write2_element_a,reject_net_write2_wasm.
netShutdown(fd: i32, how: i32): i32 shuts the read side (0), the
write side (1) or both (2).
- Tests:
net_tcp_calls.
netClose(fd: i32): i32 closes the descriptor, which is gone after the
call whatever the answer.
-
Lowering: not
willreturn, because a socket the program inherited with a linger time can wait. - Tests:
net_tcp_calls,net_tcp_echo;reject_net_fd_type.
netSetOption(fd: i32, option: i32, value: i32): i32 sets one
socket option from a whitelist. Any other option is
-22, so there is no way to pass a raw level and name through.
NET_TCP_QUICKACK is Linux's, -95 elsewhere.
- Tests:
net_set_option;reject_net_set_option_wasm.
netMss(fd: i32): i32 answers the maximum segment size a connected TCP
socket sends, or a plain -1, never an errno, for every other descriptor
and every call on wasm. It never panics.
-
Lowering: Linux's
TCP_INFOtcpi_snd_mss, orTCP_MAXSEGwhere that is not above0and on Darwin. -
Tests:
net_mss;reject_net_mss_arity,reject_net_mss_type;net_mssintests/run.jsfor wasm32 and WASI.
unixListen(path: string, backlog: i32, flags: i32): i32 answers a
listening Unix-domain stream socket at path, whose
file only its owner may connect to (0600). Its paths and refusals are under
that heading.
-
Tests:
net_unix_calls,net_unix_paths,net_unix_fds;reject_net_unix_arity,reject_net_unix_wasm.
unixListenMode(path: string, backlog: i32, flags: i32, mode: i32): i32
answers unixListen's socket with the file's mode chosen, 0 to 0o777,
set before the listen so no connect gets in under a wider one. An abstract
name is -22.
- Tests:
net_unix_paths,net_unix_calls.
unixConnect(path: string): i32 answers a new socket, non-blocking and
close-on-exec, connected to the listener at path. A full backlog is -11
on Linux and -111 on Darwin.
- Tests:
net_unix_calls,net_unix_paths;reject_net_unix_path_type.
unixAccept(fd: i32): i32 answers the next waiting connection,
non-blocking and close-on-exec; no peer address is written.
- Tests:
net_unix_calls.
unixSocketPair(kind: i32, fds: i32[]): i32 writes two connected
sockets into fds, non-blocking and close-on-exec, of kind 1
(NET_UNIX_STREAM) or 2 (NET_UNIX_SEQPACKET). Kind 2 is -95 on
Darwin.
-
Tests:
net_unix_calls,net_unix_fds;reject_net_unix_pair_element,reject_net_unix_pair_readonly.
sendFd(sock: i32, fd: i32, buf: readonly u8[], off: i32, len: i32): i32
sends bytes over the Unix socket sock with one copy of fd riding on them
(SCM_RIGHTS). A len of 0 is -22, a socket of another family
-95, and a gone peer -32, never SIGPIPE. The range is
checked.
-
Tests:
net_unix_scm,net_unix_scm_bounds;reject_net_unix_scm_arity,reject_net_unix_scm_wasm.
recvFd(sock: i32, buf: u8[], off: i32, len: i32, fd: i32[]): i32
receives bytes from the Unix socket sock, with the first descriptor the
message carried in fd[0], or -1; every further one is closed before
the answer. The one kept is close-on-exec and not made non-blocking. The
range is checked.
-
Tests:
net_unix_scm,net_unix_scm_bounds;reject_net_unix_scm_element,reject_net_unix_scm_readonly,reject_net_unix_scm_buf_readonly,reject_net_unix_scm_wasm.
peerCredentials(fd: i32, out: i32[]): i32 writes the pid, uid and gid
of the process at the other end of a connected Unix socket, as the kernel
recorded them when the connection was made, not who holds the other end
now. A socket with no peer is -107 on every platform. On Darwin the pid
is -1 where the SDK has no LOCAL_PEERPID or the kernel refuses it.
-
Tests:
net_unix_scm,caps_net_peer;reject_net_unix_peer_arity,reject_net_unix_peer_readonly,reject_net_unix_peer_element,reject_net_unix_scm_wasm.
netRingState(): i32 answers whether NET_LISTEN_RING listeners accept
into rings: 1 when they do, 0 until a listener has found rings or a
refusal, or the negative errno the kernel refused rings with (-1 when a
seccomp filter refuses io_uring_setup, -38 for a kernel without it), and
then they serve descriptors. The first listener that finds rings or a refusal
decides for the process; one whose rings fail for want of descriptors or
memory serves descriptors itself, and a later listener asks again.
It is -95 on Darwin.
-
Tests:
net_ring_calls,link/net_ring_fallback;reject_net_ring_arity,reject_net_ring_wasm.
Slot handles
A connection tcpAccept takes from a NET_LISTEN_RING listener on Linux is
a slot handle: a number at or above 2^20 that names a slot in one of the
process's io_uring rings, not a descriptor, so a process holds connections
past its RLIMIT_NOFILE. Its peer is written as zeros.
-
netRead,netWrite,netWrite2,netClose,netLocalPort,pollAdd,pollModify,pollRemoveandpollWaittake a slot handle as they take a descriptor, with the same readiness contract:-11untilpollWaitreports it, then progress. A slot is in one loop at a time: apollAddto a second loop answers-17until the first loop removes it or is closed. -
netShutdownshuts a slot down through the ring, behind the bytes already written: the peer reads them and then the end of the stream. After a shut writing side (how1 or 2)netWriteanswers-32; after a shut reading side (how0 or 2)netReadanswers what is queued and then0. Ahowpast 2 answers-22. -
netMssanswers-1: the ring has no route toTCP_MAXSEG. -
netSetOptionanswers-95: the slot's socket belongs to the ring. Every other call that takes a descriptor (sendFd,peerCredentials, FFI) answers-9, and so does every call on a handle after itsnetClose. -
tcpAcceptanswers-23when every slot its listener may hold is taken: up to 64 rings, each of as many slots as the soft descriptor limit, at most 16,384. A slot freed bynetClosetakes a waiting connection again. -
A ring listener whose accept finds memory short (
-12,-105) answers that errno fromtcpAcceptonce, as a descriptor does, pauses accepting for 10 ms, doubling to 640 ms while it recurs, and then accepts again. Waiting connections stay in the listener's backlog meanwhile. Any other errno but-11,-4,-103or-71means the listener is broken: it stops accepting andtcpAcceptanswers that errno from then on. Close the listener and listen again. -
A slot holds at most 256 KiB of bytes the kernel has not yet sent: a
netWritepast that copies what fits, and a full slot answers-11until the peer reads. Its received bytes queue up to the same bound before the runtime stops receiving for it. -
A slot whose connection has failed, a peer's reset or a failed send, answers
netWriteandnetWrite2with the errno,-104for a reset.netReadanswers it once the queued bytes are read, and on every read after, where a descriptor answers it once and then0. -
Where the kernel refuses rings, the listener serves descriptors and
netRingStateanswers why; a program written against the readiness contract runs the same either way. -
Tests:
link/net_ring_beyond_limit,link/net_ring_full,link/net_ring_sink,link/net_ring_flood,link/net_ring_halfclose,link/net_ring_starve,link/net_ring_reset,link/net_ring_transient,link/net_ring_fallback.
What tcpListenFlags's flag 1 does on each platform:
| Platform | Flag 1 |
|---|---|
| Linux | Listeners that share the port each get new connections, spread by a hash of the connection's 4-tuple; only listeners of the same user may share |
| Darwin | Listeners share the port, but one of them takes the connections: the flag buys a restart without a gap rather than a spread; flag 4 is -95 |
wasm32, --profile wasi |
Refused at compile time with NL2404, as every nish:net call is |
Errors are numbers. Every call answers an i32: >= 0 is success (a
descriptor, a count, 0), and a negative number is the failure's errno,
negated. The codes a loop branches on are Linux's numbers on every platform:
-11 would block, -95 unsupported, -32 the peer has gone, -104 the
connection was reset, -98 the address is in use, -111 the connection was
refused, -110 it timed out, -88 not a socket and -22 a bad argument; the Darwin runtime
translates its own. Any other failure is the host's -errno
(-9 for a descriptor that is not open, on both). -95 is what tcpAccept
answers for a datagram socket, on both (net_udp_calls). A number rather than a
Result, because no builtin answers one and a failure here is routine, and
because a number allocates nothing. It follows the shape signalFd set, where
a negative i32 is a failure, and adds the errno to it: signalFd and
readSignal answer a plain -1 for any failure (and readFileBytesSync
answers null), while a nish:net call says which failure it was, so a loop
tells "would block" from "the peer has gone" without asking again. No
nish:net call touches the arena (net_tcp_echo serves two hundred messages
in a loop whose pass scope gives back what each pass builds, and prints flat
when that moved the arena no further than one message did).
Socket options
netSetOption's option is one of five ids nish:net exports, each an i32
and each a global too. They are the runtime's own numbers, not the kernel's,
so a program reads the same on Linux and Darwin, and the runtime maps each to
its level and name; there is no raw passthrough, because a level, a name and a
length the program spells is an untyped call into the kernel.
| Id | Value | Sets | value |
|---|---|---|---|
NET_TCP_NODELAY |
1 |
IPPROTO_TCP, TCP_NODELAY |
1 sends a small write at once, 0 lets Nagle's algorithm batch it |
NET_TCP_QUICKACK |
2 |
IPPROTO_TCP, TCP_QUICKACK, Linux only; -95 on Darwin |
1 acknowledges at once, 0 lets the kernel delay the ACK. Linux does not keep the mode: it leaves it on its own, so a server that wants one sets it again after every response |
NET_SO_KEEPALIVE |
3 |
SOL_SOCKET, SO_KEEPALIVE |
1 probes an idle connection |
NET_SO_SNDBUF |
4 |
SOL_SOCKET, SO_SNDBUF |
bytes; Linux doubles the value it stores |
NET_SO_RCVBUF |
5 |
SOL_SOCKET, SO_RCVBUF |
bytes; Linux doubles the value it stores |
An id is read as a value, as platform is: it folds to its number in the IR,
cannot be assigned, and is not a module constant, so a
module constant cannot be initialised from one. A local of the same name
shadows the global spelling.
netMss's -1 is one answer, not an errno, because a program does the same
thing for every reason there is no MSS: it uses a size of its own. That is also
why it compiles on wasm when netSetOption is refused there: a socket option a
program sets is an effect a wasm build cannot have, while a size it reads has
an honest answer everywhere, "none".
The address form
An address is 18 bytes of a u8[]: the 16 bytes of an IPv6 address, with
an IPv4 address written as the mapped ::ffff:a.b.c.d, then the port,
big-endian, so 127.0.0.1:8080 is ten zeros, 255 255, 127 0 0 1, 31 144
(net_tcp_calls). One form for both families, in memory the caller owns, so a
server that accepts a million connections allocates nothing to learn who they
are from.
Both families on one socket. net_dual_stack listens on "::", and the
net_ block of tests/run.js connects to it from Node over 127.0.0.1 and
then over ::1: the first peer arrives as ::ffff:127.0.0.1 and the second as
::1, each with Node's own port in the last two bytes, and both are echoed.
Its UDP socket, bound to "::" too, sends itself a datagram with each ECN mark
to its IPv4 address and to its IPv6 one, and reads every mark back from
meta[1] over both. That needs a host with IPv6: where ::1 cannot be listened
on, "::" is the IPv4 fallback above, and the check says so and passes rather
than counting a skip. CI's ubuntu-latest has IPv6 and runs it.
The client half
| Function | Takes | Gives | Effect |
|---|---|---|---|
tcpConnect |
readonly u8[] |
i32 |
write |
connectResult |
i32 |
i32 |
write |
tcpConnect(addr: readonly u8[]): i32 makes a new TCP socket,
non-blocking and close-on-exec, that starts connecting to the
address in addr, and answers its descriptor at once,
while the connection is still being made; the loop reports it writable when
that has finished. Any other failure closes the socket and is answered
instead of it.
-
Tests:
net_tcp_connect,net_tcp_connect_import;reject_net_connect_element,reject_net_connect_arity.
connectResult(fd: i32): i32 answers how the connection went, asked once
fd is writable: 0, or its failure from SO_ERROR, such as -111 for a
port nobody listens on. The kernel clears the failure as it is read.
-
Tests:
net_tcp_connect,net_tcp_connect_import;reject_net_connect_wasm.
A connection is made the way everything else here waits: in the loop. Linux
answers EINPROGRESS even over loopback, so a program asks for writability and
not for an answer from the call. Neither call waits, so both are willreturn,
and neither allocates. net_tcp_connect is a Nish client and a Nish server in
one loop: two exchanges of 1 and 200 messages come back byte for byte with the
arena where it stood, and a port that was bound and closed again answers -111
and then 0. The net_ block of tests/run.js runs it under a timeout, so a
lost wake fails the run rather than hangs it.
const addr: u8[] = new Array<u8>(18);
netAddress(addr, "127.0.0.1", 8080);
const fd = tcpConnect(addr); // the descriptor, still connecting
pollAdd(loop, fd, 2, 1); // writable once it has finished
// ... pollWait until token 1 comes back ...
if (connectResult(fd) === 0) {
pollModify(loop, fd, 1, 1); // connected: now wait to read
}
UDP
| Function | Takes | Gives | Effect |
|---|---|---|---|
udpBind |
string, i32, i32 |
i32 |
write |
udpSendTo |
i32, a range, to, segment, ecn |
i32 |
write |
udpRecvFrom |
i32, a range, from, meta |
i32 |
write |
udpBind(host: string, port: i32, flags: i32): i32 answers a datagram
socket, non-blocking and close-on-exec, bound as tcpListen's is, "::" and
its IPv4 fallback included. Flag 1 is SO_REUSEPORT, flag 2 is
UDP_GRO, and flag 4, with flag 1 and a group size N from 1 to 255 in
bits 8 to 15, steers QUIC: a classic-BPF reuseport program hands each
datagram to the port's socket number (byte 0 of the Destination Connection
ID) mod N, in bind order (docs/h3-multicore.md). A refused
attach closes the socket and answers the errno. Flag 4 without 1, with
2 or with N 0, N without 4, and any other bit are -22; flags 2 and
4 are -95 on Darwin.
-
Tests:
net_udp_calls,net_udp_echo,net_udp_offload,net_dual_stack,net_quic_prefork;reject_net_udp_wasm,reject_net_udp_parallel.
udpSendTo(fd: i32, buf: readonly u8[], off: i32, len: i32, to: readonly u8[], segment: i32, ecn: i32): i32
sends buf[off, off + len) to to, cut by the kernel into datagrams of
segment bytes when that is above 0 (GSO), with ecn as the IP
header's two ECN bits. The range is checked.
-
Lowering: one
sendmsg. Notwillreturn: on a blocking socket the program inherited it waits. -
Tests:
net_udp_calls,net_udp_echo,net_udp_offload,net_udp_bounds;reject_net_udp_send_element,reject_net_udp_arity,reject_net_udp_task.
udpRecvFrom(fd: i32, buf: u8[], off: i32, len: i32, from: u8[], meta: i32[]): i32
receives the next datagram into buf[off, off + len), cut to len, with its
sender in from, its GRO segment size in meta[0] and its ECN bits in
meta[1]. It answers -11 when none is waiting. The range is
checked.
- Lowering: one
recvmsg. Notwillreturn, asudpSendTo. -
Tests:
net_udp_calls,net_udp_echo,net_udp_offload,net_udp_bounds;reject_net_udp_readonly,reject_net_udp_meta_readonly,reject_net_udp_meta_element,reject_net_udp_element,reject_net_udp_region.
A datagram carries no stream, so an echo is udpRecvFrom and then udpSendTo
back to from, which is already an address; the net_ block of
tests/run.js drives net_udp_echo from Node's dgram, two hundred
datagrams in a round, and requires every byte back and the arena flat. An
empty datagram is a datagram: udpRecvFrom answers 0 for it, which is how
net_udp_echo ends a round. A socket that is IPv4 — bound to an IPv4 host, or
to "::" on a kernel without IPv6 — sends to a mapped ::ffff:a.b.c.d
address as IPv4 (net_udp_calls).
Segmentation offload. On Linux, one udpSendTo of 4,800 bytes with
segment = 1200 leaves as four datagrams of 1,200, which a Node dgram socket
receives one by one. A socket bound with flag 2 receives what one such send
left as one datagram of up to 64 KB with meta[0] == 1200, while a socket
without it receives the four apart, each with meta[0] == 0
(net_udp_offload, run by the net_ block). The kernel coalesces only
datagrams of one flow, and only those that arrive after the option is set,
which is why it is a flag of udpBind; on loopback it coalesces what left as
one GSO send, not datagrams sent one by one. The ECN bits a send asks for
arrive in the receiver's meta[1] (net_udp_offload). Measured on loopback,
Nish sender to Nish receiver, 1,200-byte datagrams: 459,000 a second at one
per call, and 1,760,000 a second sent with segment = 1200 over 49,200-byte
sends.
Darwin has none of the three. There udpSendTo with a segment or an
ecn above 0, and udpBind with flag 2, answer -95, and meta is
always 0, 0. A meta[1] of 0 is also what a datagram that really is not
ECN-capable reads as, so a receive on Darwin cannot say ECN is unsupported: a
program learns that from the -95 a send with an ecn above 0 answers, and
on Darwin it reads every arrival as Not-ECT. CI's runtime-net (macos-latest)
job runs that branch: the net_ sections there take the -95 a GSO send and a
GRO bind answer, and fall back to one datagram a send (net_loopback,
net_relay), and net_udp_offload requires the -95 of GSO, the GRO bind and
each ECN mark, with nothing sent by them while a plain send still arrives. On Linux no UDP call answers -95, and the code is pinned there
by tcpAccept on a datagram socket instead, above.
The readiness loop
| Function | Takes | Gives | Effect |
|---|---|---|---|
pollCreate |
nothing | i32 |
write |
pollAdd |
i32, i32, i32, i32 |
i32 |
write |
pollModify |
i32, i32, i32, i32 |
i32 |
write |
pollRemove |
i32, i32 |
i32 |
write |
pollWait |
i32, i32[], i32 |
i32 |
write |
pollCreate(): i32 answers a loop: a descriptor, close-on-exec, that
watches others.
- Lowering: epoll on Linux, kqueue on Darwin.
- Tests:
net_loop_calls,net_loop_two;reject_net_poll_wasm.
pollAdd(loop: i32, fd: i32, events: i32, token: i32): i32 watches fd
for events (1 readable, 2 writable) and reports it under token, an
i32 of the program's choosing. It answers -17 for a descriptor the loop
already watches.
-
Tests:
net_loop_calls,net_loop_two;reject_net_poll_arity,reject_net_poll_parallel.
pollModify(loop: i32, fd: i32, events: i32, token: i32): i32 sets new
events or a new token for a watched descriptor. It answers -2 for one the
loop does not watch.
- Tests:
net_loop_calls,net_loop_two.
pollRemove(loop: i32, fd: i32): i32 stops watching fd. It answers
-2 for one the loop does not watch. Closing a descriptor stops the watching
too.
- Tests:
net_loop_calls,net_loop_two.
pollWait(loop: i32, ready: i32[], timeoutMs: i32): i32 waits until a
watched descriptor is ready or timeoutMs passes, forever for a negative
one, and writes a token and events pair per ready descriptor into ready,
4 added for a hang-up or an error. It answers the count, at most 64, or
0 on a timeout.
- Lowering: not
willreturn, because it can wait forever. -
Tests:
net_loop_calls,net_loop_two;reject_net_poll_readonly,reject_net_poll_element,reject_net_poll_task,reject_net_poll_region.
Level-triggered. A descriptor that is still ready when the program waits
again is reported again, so a loop that handles one pair of several, or leaves
a datagram unread, loses nothing: net_loop_calls sends a datagram and waits
twice, and both waits report it. The same is why the 64 a call reports at most
is a bound on a stack array rather than on the program: the rest are still
ready at the next wait. A UDP socket is always writable, and on Linux one whose
both sides are shut is readable, writable and hung up, 7 (net_loop_calls).
Nothing allocates; ready is the caller's.
A signal wakes the loop. The descriptor signalFd() answers is a
descriptor like any other, readable when SIGTERM or SIGINT arrives, so a server
adds it to its loop and shuts down when its token comes back: readSignal
then answers the number without waiting. A signal that interrupts the wait
itself makes pollWait answer 0, as a timeout does, rather than wait on
past the deadline it was given; the loop comes round and the next wait finds
the signal descriptor ready (tests/runtime-test.c). Once the process has
opened a NET_LISTEN_RING listener with rings, for the rest of its life,
closed or not, the rings' own work interrupts the wait too, so an interrupted wait goes on to its deadline and only the signal
descriptor reports a signal. net_loop_two watches
two UDP sockets and signalFd() in one loop, and the net_ block of
tests/run.js drives it from Node: it writes to one socket and waits for the
line that says that socket woke the loop before it writes to the other, in both
orders, and then sends SIGTERM, which wakes the loop through the signal
descriptor; the program reads 15, prints flat because the arena stood
where it did at the top of every pass, and exits 0.
const loop = pollCreate();
pollAdd(loop, server, 1, 1);
pollAdd(loop, signalFd(), 1, 2);
const ready: i32[] = new Array<i32>(32);
let running = true;
while (running) {
const n = pollWait(loop, ready, -1);
for (let k = 0; k < n; k++) {
if (ready[2 * k] === 2) {
running = false; // SIGTERM or SIGINT
} else {
serve(server); // until it answers -11
}
}
}
Darwin is kqueue, with a filter per direction rather than a set of events,
and the runtime builds epoll's answers from the filters: pollAdd of a
descriptor already watched is -17, pollModify and pollRemove of one not
watched are -2, a descriptor added watching nothing holds a disabled read
filter so the loop still knows it, and a descriptor both readable and writable
is one pair, 3, folded from its two filters' records. Three differences
stay. The pollAdd that answers -17 leaves the descriptor, both ways, under
the token it gave rather than the old one. A descriptor watching nothing is
never reported: epoll still reports its hang-up or error as 4, but a disabled
filter reports nothing. And a peer's half-close is 1 on Linux but 5, 1|4,
on Darwin: kqueue marks the read filter with end-of-file as soon as the FIN
arrives, and the runtime reads end-of-file as 4, where epoll sets its
hang-up bit only once both directions are shut. CI's
runtime-net (macos-latest) job runs net_loop_calls and net_loop_two on a
mac, as it runs every net_ section.
Unix-domain sockets
A Unix-domain socket is named by a path rather than a host and port, and
reaches only processes on the same machine. Once made, it is a descriptor like
any other: netRead, netWrite, netShutdown, netClose and the readiness
loop take it as they take a TCP one, and a peer that has gone is -32 here
too. There is no address form: unixAccept writes no peer.
-
The path. It is bytes up to the first NUL, so a NUL inside it is
-22, as an empty path is. It must fitsun_pathwith its NUL: 107 bytes on Linux, 103 on Darwin, and a longer one is-36before a socket exists, rather than the kernel's cut-short name binding a file the program never named. A relative path is the working directory's, and the limit is on the string as written, not the absolute path it resolves to. -
Abstract names. A path that starts with
@is Linux's abstract namespace: the rest, at most 107 bytes, is the name, nothing is made on the file system and nothing is left behind when the socket closes. A bare@is-22. Darwin has none, and there every@path is-95. A file whose name starts with@is spelled./@name, which is a relative path like any other. -
The file's mode. A pathname listener's file is
0600forunixListen, so only its owner may connect, andunixListenMode'smodeotherwise. The runtime sets it between the bind and the listen, while every connect is still refused, so the umask's mode is never one a client gets in under. A connect needs write permission on the file, and search permission on every directory above it, which is the real boundary: put the socket in a directory only the users who may connect can reach. -
A path that exists. A bind never replaces a file. A path that exists is
-98, whatever is there, and flag1,NET_UNIX_UNLINK_STALE, clears only a socket file nothing listens on, the one a server that died left: the runtime connects to it, and only a refusal makes it unlink the file and bind again. On Linux a live listener, even one whose backlog is full, stays-98; a regular file, a directory or a link is-17and left alone. On Darwin a connect to a listener whose backlog is full is refused just as one to a dead file is, so the probe cannot tell them apart: there a socket file in the way is-95and left alone, and the flag never unlinks one; remove a file the program knows is its own withunlinkSync. The probe is a connection the live listener sees close at once. Between the probe and the unlink another process can bind the path, which is why the directory must be the program's own (docs/security/runtime.md, RT-34). -
Cleanup. Closing a listener does not unlink its file:
unlinkSyncit when the server stops, or let the next start's flag1clear it. AunixListenthat fails after its bind closes the socket and unlinks the file it made, so no failure leaves a descriptor or a file behind (net_unix_fdscounts the descriptors around every failure, a chmod made to fail after the bind among them, and checks that none of the paths it named is left on disk). -
Pairs.
unixSocketPairmakes two sockets already connected to each other.NET_UNIX_STREAMis a byte stream, as a connection is;NET_UNIX_SEQPACKETkeeps each write one message, so a read returns one message, never two run together (net_unix_calls). Darwin has no Unix seqpacket, and kind2there is-95. -
Descriptors and credentials. The descriptor
recvFdgets for onesendFdsent is a new descriptor for the same open file, with its offset and status flags shared, asdupgives.recvFdkeeps the first descriptor of a message and closes every other one before it answers, so a peer cannot leave this process holding descriptors it never asked for (docs/security/runtime.md, RT-37). The one it keeps is close-on-exec, so a laterspawndoes not hand it on, but it is not made non-blocking:netReadon a blocking one waits, so set what the program needs on it before it joins a loop.peerCredentialsnames the process at the other end as the kernel recorded it at the connect; a peer that has changed its uid since, or passed the socket on, still reads as it was then (RT-39). On a TCP socket, or any socket notAF_UNIX, the three are-95, and on a descriptor that is not a socket-88, both before anything is sent or asked (net_unix_scm).
| Constant | Value | Meaning |
|---|---|---|
NET_UNIX_UNLINK_STALE |
1 |
unixListen's flag: clear a socket file nothing listens on |
NET_UNIX_STREAM |
1 |
unixSocketPair's byte-stream kind |
NET_UNIX_SEQPACKET |
2 |
unixSocketPair's message kind; -95 on Darwin |
NET_UNIX_UNLINK_STALE and NET_UNIX_STREAM are both 1, as
NET_TCP_NODELAY is: the constants are plain i32s, so the checker cannot
tell one passed where another belongs. Name the one the call takes.
On Darwin the calls are the same C under #if, with seven differences: no
abstract names (-95), no seqpacket pair (-95), flag 1 meeting a socket
file is -95 rather than a probe, a connect to a full backlog is -111
rather than -11, fcntl after accept and socketpair where Linux
passes SOCK_NONBLOCK | SOCK_CLOEXEC, fcntl after recvmsg where Linux
passes MSG_CMSG_CLOEXEC, so a child another thread spawns between the two
inherits the descriptor recvFd keeps (RT-38), and peerCredentials reading
getpeereid and LOCAL_PEERPID, its pid -1 where that refuses, where
Linux reads SO_PEERCRED. A listener has no peer: Linux answers its own
process's credentials there, and what Darwin answers is not yet seen on a
mac. CI does not run them on a mac yet.
Bounds
netRead, netWrite, netWrite2 (each of its two ranges), udpSendTo, udpRecvFrom and fileReadAt check 0 <= off <= off + len <= buf.length in the compiled code, before the call, the way dst.set(src, at) checks the range it
writes, and a range outside it panics with the same words, slice out of range: [4, 9) of length 8, exit 1 (net_tcp_bounds, net_udp_bounds, net_write2_bounds, file_read_at_bounds). A negative
off or len fails it. A peer, out, to or from
array shorter than 18 bytes, or a meta shorter than 2, is not a panic but
-22, from the runtime, because the length is the argument's
property rather than the program's arithmetic. So is a ready shorter than 2,
and pollWait never writes past ready.length, so it has no range to check.
Types, threads and wasm
-
Bytes only. Each buffer is a
u8[], ascrypto.getRandomValues's is: a socket carries bytes, and a wider element would make the byte order on the wire a fact about the machine. Anything else is`netRead` fills a `u8[]`, got i32[](NL2402;reject_net_element) for a buffer the call writes, and`netWrite` sends from a `u8[]`, got string(NL2405;reject_net_write_element,reject_net_udp_send_element) for the one it reads.udpRecvFrom'smetaholds numbers rather than bytes, a segment size up to 65,535 among them, so it is ani32[], and anything else is`udpRecvFrom` fills an `i32[]`, got u8[](NL2406;reject_net_udp_meta_element), and so ispollWait'sready, whose tokens are the program's owni32s (`pollWait` fills an `i32[]`, got u8[],reject_net_poll_element), and so isunixSocketPair'sfds(`unixSocketPair` fills an `i32[]`, got u8[],reject_net_unix_pair_element), and so arerecvFd'sfdandpeerCredentials'sout(reject_net_unix_scm_element,reject_net_unix_peer_element). A path is astring, and anything else is refused (reject_net_unix_path_type). A descriptor, an offset, a length, a port, a backlog,how, the flags, a segment size, the ECN bits, a loop, the events, a token, a timeout, a socket option and its value, a pair kind and a mode arei32in both number modes, and anf64is refused rather than converted (reject_net_fd_type). -
What is written.
netAddress'sout,tcpAccept'speer,netRead'sbuf,udpRecvFrom'sbuf,fromandmeta,pollWait'sready,unixSocketPair'sfds,recvFd'sbufandfdandpeerCredentials'soutare written, so areadonlyarray there is refused as a store through it is (reject_net_readonly,reject_net_accept_readonly,reject_net_udp_readonly,reject_net_udp_meta_readonly,reject_net_poll_readonly,reject_net_unix_pair_readonly,reject_net_unix_scm_readonly,reject_net_unix_scm_buf_readonly,reject_net_unix_peer_readonly), a parameter one of them fills is neverreadonlyin the IR, andnetWrite'sbuf,netWrite2'saandb,udpSendTo'sbufandtoandsendFd'sbufmay bereadonly(net_udp_calls,net_write2_bounds). -
Threads. Every call changes the kernel's state of a socket, which is a
write its caller can see, and a loop is a socket's state as much as its
bytes, so none may be made in a
scope()task (reject_net_task,reject_net_udp_task,reject_net_poll_task) or a parallel body (reject_net_parallel,reject_net_udp_parallel,reject_net_poll_parallel), and a call that writes an array is refused in a scope's region on one a task may read, asfillis (reject_net_region,reject_net_udp_regionformetaandreject_net_poll_regionforready). -
On wasm. A
wasm32target and thewasiprofile have no sockets for these to reach, andruntime-net.cis empty there, so every call is refused by the checker with the host builtins' words:`tcpListen` reaches the operating system, and a wasm32 build has none to reach(NL2404;reject_net_wasm,reject_net_import_wasm,reject_net_udp_wasm,reject_net_poll_wasm,reject_net_reuseport_wasm,reject_net_set_option_wasm,reject_net_unix_wasm). The freestanding--profile wasmlink, which takes IR compiled for the host, has noruntime-net.ceither, so a program that calls one fails there at the link, on the undefinednish_symbol. The one exception isnetMss, which compiles under awasm32target and thewasiprofile to the constant-1, its descriptor still evaluated, with no runtime call (net_mssintests/run.js). -
Not yet. No
nish:netcall resolves a name (reject_net_unknown_export): the stub resolver is the library modulenish/net/dns, which drives these calls on the program's own loop. There is nosendmmsgorrecvmmsg, no edge-triggered mode and no timer butpollWait's timeout. A program that does not want a loop can still wait for a connection, a byte or a datagram by calling again on-11, asnet_tcp_echoandnet_udp_echodo.
The TypeScript reading (wp33-round-trip.md §2) is
class C, as signalFd's is: Node's sockets are asynchronous only, and a
socket becomes ready only to Node's event loop, which a program that owns its
loop never returns to. runtime/nish.d.ts declares nish:net and the globals,
so npm run check types these programs, and under runtime/nish.mjs each
function throws `netAddress` has no synchronous reading under Node: a socket is ready only to the event loop, which a program that owns its loop never returns to (docs/wp33-round-trip.md) rather than answer -11 forever (the
net_ block of tests/run.js, for netAddress, tcpConnect, udpBind and
pollCreate). The option ids are plain numbers, so they read the same
under Node as natively, and netSetOption throws rather than set one
(net_set_option in tests/run.js). netMss throws too rather than answer -1: Node's
net.Socket has no MSS to read, and a -1 would be a guess that the program
keeps (net_mss in tests/run.js). The same holds for the Unix-domain calls: the
NET_UNIX_ constants read natively, and each of the five calls throws
(net_unix in tests/run.js). In a browser there are no sockets at all.
A loop has no reading either: the one a Node program has is Node's own, which
it returns to between callbacks, and pollWait is a program that never
returns to it.
nish:term: terminal input
What a program needs to run a full-screen terminal interface: read a key the
moment it is pressed, wait for one with a timeout that doubles as a frame
clock, and ask the terminal whether it is one, how big it is, and to stop
echoing and line-editing. None of it needs the net capability that a
pollAdd(0) and netRead(0, …) would carry; the six functions carry
tty instead. They are C in runtime/runtime-term.c, a
translation unit of its own with its own .text* ceiling, which section GC
leaves out of a program that calls none of them. Each is exported by
nish:term and is a global too, as every nish: export is. Linux and Darwin
only, and WASI as below.
Reference: nish:term.
| Function | Takes | Gives | Effect |
|---|---|---|---|
readStdin |
u8[], i32, i32 |
i32 |
write |
waitStdin |
i32 |
boolean |
write |
isTerminal |
i32 |
boolean |
write |
terminalSize |
nothing | i32[] |
write |
setRawMode |
boolean |
i32 |
write |
setTerminalRestore |
string |
i32 |
write |
readStdin(buf: u8[], off: i32, len: i32): i32 reads at most len bytes
of stdin into buf[off, off + len), without waiting. It answers the
count, 0 at the end of the stream, or -11 when nothing is waiting. The
range is checked, and the buffer is a writable u8[].
-
Tests:
term_calls,term_import;reject_term_read_arity,reject_term_read_i32_buffer,reject_term_read_readonly.
waitStdin(timeoutMs: i32): boolean waits until stdin is readable, its
end and an error included, or timeoutMs passes, for ever for a negative
one. It answers false when the time ran out or a signal ended the wait
early, but not for a Ctrl-Z and fg once setRawMode(true) has caught
SIGTSTP, after which the wait goes on for what is left. Before raw mode Linux
goes on with what is left and Darwin ends the wait.
- Lowering: not
willreturn. - Tests:
term_calls,term_import,term_pty;reject_term_wait_string.
isTerminal(fd: i32): boolean answers whether the descriptor is a
terminal.
- Lowering:
isatty. - Tests:
term_calls,term_pty;reject_term_is_terminal_arity.
terminalSize(): i32[] answers a fresh [columns, rows] of the terminal
on stdout, or on stderr when stdout is not one; [-25, 0] for "not a
terminal".
-
Tests:
term_calls,term_pty,mem_terminal_size_scope;reject_term_size_arity.
setRawMode(on: boolean): i32 puts stdin's terminal in raw mode (true)
or back as it was (false). It answers 0, or -25 when stdin is not a
terminal. setRawMode(false) is always safe.
-
Tests:
term_calls,term_pty;reject_term_raw_number,reject_term_unknown_export.
setTerminalRestore(bytes: string): i32 keeps a copy of bytes for the
runtime to write to the terminal on stdout as the program ends, replacing
what it kept before. It answers 0, -25 when stdout is not a terminal and
-7 (E2BIG) past NISH_TERM_RESTORE_MAX, 4096 bytes, both keeping what was
there. When they go out is under Raw mode.
-
Tests:
term_restore;reject_term_restore_arity,reject_term_restore_number,reject_term_restore_wasm.
Errors are numbers, in nish:net's shape: >= 0 is success and a negative
number the failure's errno, negated. The two a program branches on are the
same number on every platform: -11, nothing is waiting, and -25, not a
terminal. Any other is the host's own.
stdin is never made non-blocking. O_NONBLOCK belongs to the open file
description, which a shell shares with every program it starts, so a program
that set it and then crashed would leave the shell's own stdin non-blocking.
readStdin asks select with a timeout of 0 before it reads instead, and so
does waitStdin: Darwin's poll does not support devices, and answers a
terminal at once whether a key is waiting or not.
The loop. waitStdin is the frame clock: a game waits up to one frame for a
key, reads what came, advances the world and draws, so the wait costs nothing
when a key arrives early and the frame rate holds when none does.
import { readStdin, setRawMode, waitStdin } from "nish:term";
export const main = (): number => {
setRawMode(true);
const keys: u8[] = new Array<u8>(16);
let running = true;
while (running) {
if (waitStdin(16)) {
const n = readStdin(keys, 0, keys.length);
if (n === 0 || (n < 0 && n !== -11)) {
running = false; // stdin ended, or reading it failed
}
for (let i = 0; i < n && i < keys.length; i++) {
if (keys[i] === 113) {
running = false; // q
}
}
}
// advance and draw one frame
}
setRawMode(false);
return 0;
};
Raw mode
Raw mode is cfmakeraw's input side without its output side. Echo, line
editing, CR-to-NL translation, flow control (Ctrl-S, Ctrl-Q) and parity
stripping are off, bytes are 8 bits wide, and a read answers as soon as one byte
is there, so a key arrives as its bytes — an arrow as the three of ESC [ A.
Two things stay as they were:
-
ISIGstays on. Ctrl-C is still SIGINT and ends the program, Ctrl-\ is still SIGQUIT and Ctrl-Z still stops it; none of them arrives as a byte (3, 28, 26). A program that wants Ctrl-C as a key asks for SIGINT throughsignalFdand handles it in its loop. Keeping it means a program that is stuck in a loop can always be ended from the keyboard. -
OPOSTstays on, so"\n"still moves to the start of the next line and output needs no"\r\n".
The terminal is always put back. Each setRawMode(true) that enters raw
mode saves the termios that was there, and it comes back on
setRawMode(false); when main returns or process.exit is called (an
atexit handler); on every panic, a failed check included, whose path ends in
_exit and so is given its own hook; and on every signal whose default action
ends the program, SIGKILL aside: SIGINT, SIGTERM, SIGQUIT and SIGHUP, SIGPIPE
from a reader that went away, SIGALRM, SIGUSR1, SIGUSR2, SIGPROF, SIGVTALRM,
SIGXCPU, SIGXFSZ and SIGSYS, and the faults a program dies of, SIGILL, SIGTRAP,
SIGABRT, SIGFPE, SIGBUS and SIGSEGV (a stack overflow, or a fault in C code a
program links). For each a handler is installed only when the program left it
at its default action: it restores the terminal, puts the default back and
raises the signal again, so the program still dies of it and its parent sees
that (exit status 143 for SIGTERM in term_pty). The handlers run on a 64 KiB
alternate stack of the thread that entered raw mode, or on the one that thread
already had, so a stack overflow there restores the terminal too. The mode is
marked raw once the switch is made, so a signal that ends the program while
TCSAFLUSH drains the output, or while SIGTTOU holds it stopped in the
background, leaves the terminal as it was. A program that
took SIGTERM or SIGINT through signalFd keeps it, and puts the terminal back
itself, or by returning. SIGKILL leaves raw mode behind, and so does a stack
overflow on a thread other than the one that called setRawMode(true); stty sane undoes either. SIGTSTP, Ctrl-Z, is caught the same way when it is at its
default, because no shell can be counted on to: bash puts its own termios back
while a job is stopped but does not give the job its raw mode back on fg, and
a shell without that bookkeeping puts back nothing. The handler restores the
terminal, stops the program, and once fg or bg continues it puts the handler
back and enters raw mode again, so the terminal is as it was while the program
is stopped and raw after fg; no setRawMode on another thread runs between
the two. A program continued in the background stops again, by SIGTTOU, until
fg. A waitStdin the stop interrupts goes on waiting until its timeout,
measured from when it began. Any other blocking call goes on where the system
restarts it after a handler (SA_RESTART), and ends early where it never
does, as a pollWait does after any signal. A stop by SIGSTOP, which no program can catch,
leaves the terminal raw at a shell that puts nothing back.
Job control also stops a program started in the background
(app &) when it calls setRawMode(true), by SIGTTOU, as it stops any program
that sets the terminal, and the program enters raw mode on fg. Two threads that call setRawMode at
once take turns, so the termios saved is the one from before either.
Restore bytes go out on the same paths. The bytes setTerminalRestore
kept are written to stdout once, with write alone, before the termios comes
back, when main returns or process.exit is called, on every panic and on
each signal above whose default ends the program, raw mode or not: setting
them installs the same handlers. Ctrl-Z and setRawMode(false) do not end the
program, so they write nothing. A program sets them once it has switched the
terminal ("\u001b[?1049h" written, say) and again with each change, last
the set that undoes them all; a signal that lands while it does writes the
old string or the new one, whole. They are written at most once, however
many of those paths race. Output is not buffered, so all the program printed
reaches the terminal before them. term_restore under script(1) checks that
what follows its last line is exactly its bytes, once, after it returns,
panics and is killed by SIGTERM and by SIGSEGV, nothing after it cleared them,
and that stty -g reads as before.
The term block of tests/run.js runs term_pty under script(1), which
gives it a pseudo-terminal: it is a terminal on both descriptors with a size,
enters raw mode twice and leaves twice, each 0, and after it returns in raw
mode, panics in raw mode, and is killed by SIGTERM, and then by SIGSEGV, while
it waits in raw mode, stty -g reads what it read before. Where script is
missing or makes no terminal, that check is a counted skip that says which.
Where there is no terminal
| Host | readStdin |
waitStdin |
isTerminal |
terminalSize |
setRawMode |
setTerminalRestore |
|---|---|---|---|---|---|---|
| Linux, Darwin | as above | as above | isatty |
the ioctl | as above | as above |
| A pipe or a file | as above | as above | false |
[-25, 0] |
-25 for true, 0 for false |
-25, 0 for "", -7 past the cap |
--profile wasi |
waits for its bytes | as above | false |
[-25, 0] |
-25 for true, 0 for false |
-25, keeping nothing; 0 for "", -7 past the cap |
--target wasm32, freestanding |
refused | refused | refused | refused | refused | refused |
| Node (prelude) | throws | throws | tty.isatty |
Node's columns and rows |
throws | throws |
A browser (nish/runtime/browser) |
throws | throws | false |
[-25, 0] |
throws | throws |
WASI has read and poll on stdin and no terminal interface. There
readStdin reads without asking poll first, because Node's WASI answers a
zero-timeout poll of stdin with "nothing waiting" while bytes are, so it waits
for its bytes: call waitStdin first, as the loop above does. For the same
reason a waitStdin there waits at least 10 ms: Node's WASI counts a shorter
wait from before it looks at stdin, and on a busy machine answers "nothing
waiting" before it has looked. The checker
accepts the six there, unlike nish:net's, and refuses them on a
freestanding --target wasm32 build, whose runtime has no stdin
(reject_term_wasm). Under Node,
stdin is read only by the event loop, which a program that owns its loop never
returns to, and Node's own raw mode turns Ctrl-C into a byte, so the four
that would need either throw, each naming itself. The term and term_wasi
blocks of tests/run.js feed term_calls and term_import over a pipe —
four bytes after a delay, a pipe held open and never written, and a thousand
bytes — natively and under WASI, and expect the same lines from both.
Arrays and strings as receivers
The reference for each member, with its parameters and its answer, is its doc comment on Array, ReadonlyArray or String in runtime/nish.d.ts; the rules are here.
| Member | On | Gives |
|---|---|---|
length |
arrays | number |
push(v) |
arrays | number |
pop() |
arrays | T |
indexOf(v) |
arrays | number |
join(sep?) |
arrays | string |
set(src, offset?) |
arrays | void |
fill(value, start?, end?) |
arrays | void |
sort(cmp) |
arrays | T[] |
map(f) |
arrays | U[] |
filter(pred) |
arrays | T[] |
some(pred), every(pred) |
arrays | boolean |
findIndex(pred) |
arrays | number |
forEach(f) |
arrays | void |
slice(start?, end?) |
arrays | T[] |
length |
strings | number |
charCodeAt(i) |
strings | number |
substring(start, end?) |
strings | string |
slice(start, end?) |
strings | string |
indexOf(sub, from?) |
strings | number |
startsWith(sub), endsWith(sub) |
strings | boolean |
String.fromCharCode(c) |
strings | string |
String.fromBytes(bytes, start, end) |
strings | string |
a.length is a number; read-only.
- Tests:
arr_length,reject_arr_length_assign.
a.push(v: T): number appends and returns the new length. An array
never grows past 2^31 − 1 elements, in either number mode: the push that
would fails as an allocation does, nish: out of memory and exit 1
(tests/link/cg_sec_push_limit).
-
Lowering: grows capacity by doubling (4 from 0) through
nish_array_grow. - Tests:
arr_push;reject_arr_push_type(Cannot push string onto i32[]).
a.pop(): T removes and returns the last element. An empty array
panics and exits 1 (index out of range: 0 >= 0) — there is no
undefined to return; behind a test that the array holds an element, such
as if (a.length > 0), the check is proven away (panics_pop). The
capacity is untouched, so the next push reuses the storage.
- Tests:
arr_pop_index;reject_arr_pop_arity.
a.indexOf(v: T): number answers the first index whose element is
=== v, or -1: strings by content, classes / interfaces / arrays by
identity, floats with fcmp oeq (a NaN element is never found, as in
JavaScript).
- Lowering: a scan in the emitted code.
- Tests:
arr_pop_index;reject_arr_index_of_type.
a.join(sep: string = ","): string is for string[] only
(`join` requires string[], got i32[], reject_arr_join_elem). Element
conversion would allocate per element, which is the quadratic shape join
exists to replace (wp14-selfhost.md §3). Like a
concatenation, its result is held to 2^31 − 1 bytes: a longer one fails as
an allocation does, nish: out of memory and exit 1, before anything is
allocated (CG-3).
-
Lowering: one pass summing the lengths, one allocation, one
memcpyper part. - Tests:
arr_join,cg_sec_join_limit;reject_arr_join_elem.
a.set(src: T[], offset: number = 0): void copies every element of
src into a from offset on, with TypedArray.prototype.set's meaning.
Arrays of numbers only, of any width and ranged integers included, and
src has a's own element type. A statement. See
Bulk writes.
-
Tests:
bytes_set_disjoint,bytes_set_self,bytes_set_end,bytes_set_oob,bytes_set_overlap,bytes_set_float_oob,bytes_set_u64_oob;reject_bytes_set_string,reject_bytes_set_type,reject_bytes_set_arity,reject_bytes_set_readonly,reject_bytes_set_value.
a.fill(value: T, start: number = 0, end: number = a.length): void
stores value into every slot of [start, end), each end relative and
clamped as TypedArray.prototype.fill's are. Arrays of numbers only. A
statement. See Bulk writes.
-
Tests:
bytes_fill,bytes_fill_zero,bytes_offset_float,bytes_offset_u64;reject_bytes_fill_record,reject_bytes_fill_arity,reject_bytes_fill_value,reject_bytes_fill_index,reject_bytes_fill_value_position,reject_bytes_fill_readonly.
a.sort(cmp: (a: T, b: T) => number): T[] sorts a in place,
stably, and answers a itself, as Array.prototype.sort does: the
order is element for element Node's for any consistent comparator. cmp
answers a negative number, zero or a positive number, and only "positive or
not" is read, so -0 and NaN count as zero.
The comparator is required: a.sort() compares as strings in
JavaScript and is refused
(`sort` needs a comparator: write `xs.sort((a, b) => a - b)`, NL2620);
a second argument is refused too (`sort` takes one comparator,
NL2619). A readonly T[] is refused (Cannot `sort` through readonly),
and so is process.argv (`process.argv` is read-only, as push is).
a - b panics on i32 elements a full range apart ([2147483647, -1]);
(a, b) => (a > b ? 1 : a < b ? -1 : 0) cannot overflow.
The comparator is a compile-time function argument
(Function parameters): a top-level function, or an
arrow, which may read the const locals and parameters of the function it
is written in; an arrow reading a let is refused
(reject_arr_sort_capture_let). See Sorting.
-
Lowering: compiled to
std/sort.ts'ssortWith, a merge sort over runs of 32 sorted by insertion, instantiated once per comparator with the comparator called directly; it is copied into the module that sorts. -
Tests:
arr_sort_cmp,arr_sort_cmp_f64,arr_sort_cmp_record,arr_sort_edge,arr_sort_capture,arr_sort_overflow;reject_arr_sort_no_comparator,reject_arr_sort_arity,reject_arr_sort_readonly,reject_arr_sort_argv,reject_arr_sort_capture_let.
a.map(f: (x: T[, i: number]) => U): U[] answers a new array of
f(a[i]) (or f(a[i], i)) for each element, in order, as
Array.prototype.map gives. The callback takes one or two parameters; a
third, JavaScript's array argument, is refused
(The callback of `map` takes 3 parameters, and Nish passes at most two …,
NL2698), and so is a second argument to map itself (JavaScript's
thisArg). It is a compile-time function argument under the comparator's
rule. See Callback methods and slice.
-
Lowering: compiled to
std/array.ts'smapormapIndexed, called directly. -
Tests:
arr_map,arr_map_f64,arr_map_template;reject_arr_map_third_param,reject_arr_map_this_arg,reject_arr_map_capture_let.
a.filter(pred: (x: T[, i: number]) => boolean): T[] answers a new
array of the elements pred keeps, in order. pred answers boolean
(reject_arr_filter_not_boolean); the callback rules are map's.
- Tests:
arr_filter;reject_arr_filter_not_boolean.
a.some(pred), a.every(pred): boolean answer whether some (every)
element satisfies pred, stopping at the first that decides; an empty array
answers false (true).
- Tests:
arr_some_every.
a.findIndex(pred): number answers the first index pred accepts, or
-1. There is no find: TypeScript types its answer T | undefined
and Nish has no undefined, so xs.find(f) is refused and points here
(`find` is not supported: TypeScript types its answer `T | undefined` …,
NL2697).
- Tests:
arr_find_index;reject_arr_find.
a.slice(start: number = 0, end: number = a.length): T[] answers a copy
of a[start..end), with s.slice's rule: 0 <= start <= end <= a.length,
or the program panics with
slice out of range: [start, end) of length len and exits 1, where
JavaScript counts a negative index from the end and clamps the rest. It
takes zero to two arguments (NL2699), each a number, where s.slice takes
any integer type. It works on a readonly T[], and the copy is a mutable
T[].
- Tests:
arr_slice,arr_slice_bounds;reject_arr_slice_args.
s.length is a number, the UTF-8 byte length.
- Tests:
str_length.
s.charCodeAt(i: number): number answers the byte at i,
bounds-checked against s.length exactly as a[i] is — out of range
panics and exits 1, where JavaScript answers NaN, which number cannot
hold.
- Lowering: no call: a
load i8. - Tests:
str_bytes;reject_str_char_code_arity.
s.substring(start: number[, end: number]): string answers the bytes
of [start, end), end defaulting to s.length. Both ends are clamped
into [0, s.length] and then swapped into order, as in JavaScript, so
s.substring(5, 0) is s.substring(0, 5) and a negative offset is 0.
-
Lowering: one allocation and one
memcpy(nish_str_new). The clamp is twollvm.smin/llvm.smaxcalls per bound and is not emitted for a bound the compiler can place in[0, s.length], because a clamp cannot move a value that is already inside its range: the proof is the same one that removes a bounds check (Element access), and a literal0is proven for every string, since none has a negative length. Each bound is proven against the facts that hold where that bound is evaluated — the arguments run left to right and the receiver's length is read before either of them, so an assignment in argument 1 cannot prove argument 0 and an argument that rebinds the receiver takes the proof with it (perf_clamp_order,perf_clamp_rebind). A bound it could not prove keeps the clamp and is reported by the performance warning below. -
Tests:
str_bytes,perf_clamp_quiet,perf_clamp_order,perf_clamp_rebind;reject_str_substring_arity.
s.slice(start: number[, end: number]): string answers the same bytes,
without the clamp: start and end must satisfy
0 <= start <= end <= s.length and anything else panics with
slice out of range: [start, end) of length len and exits 1, where
JavaScript would count a negative offset from the end and answer "" for a
reversed pair. end defaults to s.length, an empty range is "", and an
in-range non-negative pair gives exactly what JavaScript's
String.prototype.slice gives.
-
Lowering: two
icmp uleagainst a cold panic block, one allocation and onememcpy. Measured 1.18x oversubstringon a lexer-shaped scan (wp15-performance.md §4). -
Tests:
str_slice,str_slice_panic;reject_str_slice_arity,reject_str_slice_type.
s.indexOf(sub: string[, from: number]): number answers the first
byte offset at or after from at which sub occurs, or -1; from
defaults to 0.
from is clamped into [0, s.length] first, as in JavaScript: a negative
start searches from 0, and a start past the end finds nothing, except that
the empty needle answers the clamped start, so s.indexOf("") is 0,
s.indexOf("", 3) is 3 and s.indexOf("", 99) is s.length. A float
start converts as JavaScript's ToIntegerOrInfinity does — a fraction
truncates and NaN is 0 — and an unsigned one is clamped from above only,
so a u64 past 2^63 is past the end.
Under Node the program is the same call to String.prototype.indexOf,
whose answers differ only where every offset does, on text that is not ASCII
(NL8001, under "Portability warnings").
-
Lowering: one
nish_str_index_ofornish_str_index_of_fromcall and no allocation: the search is the runtime's, so it is the libc's vectorised one rather than a probe per offset (WP15 §7c), and the clamp is allvm.smin/llvm.smaxpair in the caller, which a literal0does without. A loop that walks every match,at = s.indexOf(sub, at + sub.length), therefore copies nothing. -
Tests:
str_search,str_index_of_from,str_index_of_from_port;reject_str_index_of_type,reject_str_index_of_arity,reject_str_index_of_from_type.
s.startsWith(sub: string): boolean answers whether sub's bytes are a
prefix.
- Lowering:
nish_str_at. - Tests:
str_search.
s.endsWith(sub: string): boolean answers whether they are a suffix; a
sub longer than s is false.
- Tests:
str_search.
String.fromCharCode(c: number): string answers the one-byte string of
c & 0xFF, the inverse of charCodeAt. A value above 127 makes a byte that
is not valid UTF-8 on its own; nothing validates it.
- Tests:
str_search.
String.fromBytes(bytes: u8[], start: number, end: number): string
answers the bytes of bytes[start..end) as one string, where a loop of
String.fromCharCode and a join allocates once per byte. bytes is a
u8[] reached any way an array is (a local, a field, a call, a ternary, a
literal), readonly or not, since the call only reads it; anything else,
a wider element or a u8[] | null included, is
`String.fromBytes` expects an argument of type u8[], got i32[]
(NL2268). Both ends are required (NL2060) and are numbers (NL2058).
The range is s.slice's, 0 <= start <= end <= bytes.length, checked once
after all three arguments are evaluated, so an argument that grows or
shrinks the array is checked against what it left; anything else panics
with slice out of range: [start, end) of length len and exits 1.
As with fromCharCode, nothing validates the bytes as UTF-8: a multi-byte
character copied whole is that character, and a range that cuts one is bytes
that are not a valid string. Natively the string's bytes are the array's, so
write or writeFileSync sends any bytes back out unchanged
(File I/O, io_write_bytes). Under Node, where
String.fromBytes is the prelude's (runtime/nish.mjs), each byte is one
UTF-16 unit, as fromCharCode makes it, so ASCII agrees and a byte above
127 does not (a C row, wp33-round-trip.md §3.2); under
--emit-ts the bytes decode as UTF-8, a sequence that is not UTF-8 to
U+FFFD. Bytes that are not text go out through writeBytes and
writeFileBytesSync, which agree on every host. src/ does not call it
until the release that ships it (the rolling freeze).
-
Lowering: one range check, one allocation and one
memcpy(nish_str_new). -
Tests:
str_from_bytes,str_from_bytes_ways,str_from_bytes_order,str_from_bytes_lifetime,str_from_bytes_arrow,str_from_bytes_oob;reject_str_from_bytes_type,reject_str_from_bytes_elems,reject_str_from_bytes_start,reject_str_from_bytes_index,reject_str_from_bytes_arity,reject_str_from_bytes_arena,reject_deny_panics_from_bytes.
Every offset above is a byte offset, like s.length: a lexer walks bytes,
and a code-point index would need a decode per access (codePointLength in
nish/text counts code points natively, and under Node for any string that is
not built from bytes). Cutting a multi-byte
character in half is therefore possible and produces bytes that are not a
valid string (docs/wp13-differential.md notes what Node then does with them).
A numeric literal argument to push, indexOf or fill takes the element
type, so wide.push(3) on an i64[] is an i64 three
(Numeric literals).
Sorting with a comparator
xs.sort(cmp) sorts with a function the program passes. The checker
rewrites the call to sortWith(xs, cmp) from std/sort.ts, which the program
does not import: a .sort( call is what loads it. Because cmp is a
compile-time function parameter, each comparator gets its own instantiation
whose inner loop calls it directly, with no function pointer and no boxing of
its answer. The sort is stable, so for a comparator that is a consistent
total preorder there is exactly one answer and it is Node's; under Node and
--emit-ts the call is the platform's own Array.prototype.sort. A 10,000
element array of records with many ties (arr_sort_cmp_f64) prints the same
checksum natively and under Node.
The sort costs one scratch array of n elements, allocated once at full size
(new Array<T>(n) for any T, which sortWith alone may write: the copy
after it fills every slot before one is read) from the function's arena and
released before sortWith returns, so a sort leaves Arena.used() where it was
and a loop that sorts keeps its per-pass scope (arr_sort_edge). The scratch is
not on the stack: its length is the array's, known only at run time. Insertion sorts
runs of 32 from the scratch into the array, then bottom-up passes merge pairs
of runs back and forth between the two. A pair already in order (the last of
the left run not after the first of the right) is copied without a comparison
per element, so a merge pass over sorted input costs one comparison per pair.
Every move is a whole slot: an array of interface records, which is stored
inline, moves a record as one copy of its bytes (arr_sort_cmp_record), and
the sort never holds an element across a write, which would read a record half
overwritten. Every index inside the sort is proven in range by a guard beside
it, so the sort keeps no bounds check, and at -O2 the comparator is inlined
into the loops that call it.
Where JavaScript leaves the order to the implementation, so does Nish: an
array holding NaN under (a, b) => a - b, or a comparator that is not a
consistent order (one answering 1 for both (a, b) and (b, a)), comes
out a permutation of its elements, but not necessarily Node's permutation. A
program meant to agree with Node gives NaN a place of its own in the
comparator.
The idiom (a, b) => a - b is checked arithmetic on i32: two elements more
than 2^31 - 1 apart, as in [2147483647, -1], panic with
attempt to subtract with overflow. On values that may span the whole range,
compare instead of subtracting: (a, b) => (a > b ? 1 : a < b ? -1 : 0).
What differs from JavaScript is refused rather than reinterpreted: no
comparator (JavaScript would compare the elements' strings), a readonly
receiver or process.argv, and a comparator that reads a let of the
function it is written in (reject_arr_sort_capture_let). A comparator is an
arrow argument like any other, so it may read that function's const locals
and parameters (arr_sort_capture), under the rule in Function parameters.
Callback methods and slice
xs.map(f), xs.filter(f), xs.some(f), xs.every(f), xs.findIndex(f),
xs.forEach(f) and xs.slice(...) are lowered as sort is: the checker
rewrites the call to a template of std/array.ts, which a call of one of those
names loads, and the callback is a compile-time function parameter, so each
callback gets an instantiation of its own whose loop calls it directly.
xs.map(f) and map(xs, f) with map imported from nish/array are the same
call and compile to one instance (arr_map_template). The template is picked by
the callback's arity: map for (x) => ..., mapIndexed for (x, i) => ...,
and slice, sliceFrom or sliceAll by the argument count.
xs.forEach(f) answers void, so its answer cannot be bound
(reject_foreach_answer), and its callback is written to answer nothing: a
concise body such as (x) => console.log(x) or (x) => ys.push(x) is a
statement, and what it computes is dropped. It compiles to a counted loop over
the array with the callback inlined into its body, as for...of does, but the
loop stops at the length the array had when the call began, as Node does,
where for...of reads the length on every pass. With a callback that does not
resize the array, opt -O2 leaves the same instruction count for both: 18 on
the walk function measured in
IR_COOKBOOK.md. An arrow argument
never assigns a name of the function it is written in (NL2586), and a
forEach callback is no exception, so a running total is a for...of loop,
whose body may assign it; the refusal says so (reject_foreach_capture_sum):
for (const x of xs) { sum += x }.
The callback takes the element, or the element and its index, as TypeScript
allows. JavaScript passes a third argument, the array itself; Nish refuses a
callback that declares one (NL2698), since the array is the receiver the call
already names. A callback that declares no parameter, xs.map(() => 0), is
refused too, though TypeScript accepts it: the template is picked by the
callback's arity, and there is none for zero, so write (_x) => 0. The
callback rule is the comparator's: a top-level function, or
an arrow that reads only the const locals and parameters of the function it
is written in (reject_arr_map_capture_let).
Each method walks the array from index 0 and calls the callback once per
element, in order, up to the length the array had when the call began, as
Array.prototype does; some, every and findIndex stop at the first
element that decides. A callback that shrinks the array ends the walk at the
new length, where JavaScript goes on and skips the missing indices. So a
forEach callback that pushes onto its receiver does not visit what it
pushed, one that stores ahead of the walk is handed the stored value, and one
that pops stops the walk early, as Node does (arr_foreach). map and
filter answer a new array, so writing into it does not touch the receiver;
an element that is an interface record is stored inline and copied, so
filter(...)[0].id = 10 leaves the receiver's record as it was, as it would
under Node (arr_filter).
There is no find. TypeScript types xs.find(f) as T | undefined, and
Nish has no undefined: a program testing r !== null would pass tsc and
mean something else under Node. The call is refused with a message that
points to findIndex (NL2697, reject_arr_find):
const i = xs.findIndex(f); if (i >= 0) { ... xs[i] ... }.
slice panics where JavaScript clamps. The bounds rule is string
slice's: 0 <= start <= end <= xs.length, and anything else exits 1 with
slice out of range: [start, end) of length n (arr_slice_bounds). A
negative index, which JavaScript counts from the end, and an end past the
length, which it clamps, are both outside. A number bound under
--number-mode f64 is made an integer before the check, as llvm.fptosi.sat
does: a fraction is truncated toward zero (xs.slice(0.9, 2.5) is
xs.slice(0, 2)), NaN is 0, and ±Infinity or a magnitude of 2^63 or more
saturates to the i64 limits, so it fails the check and the message prints
the limit (slice out of range: [-9223372036854775808, 9223372036854775807) of length 3; arr_map_f64, arr_slice_saturate). The bounds are number
only, where string slice takes any integer type: an i64 or u32 bound is
refused (Argument 2 of `slice`: expected i32, got i64;
reject_arr_slice_i64_bound), so convert it first: with toI32(n) in the
default i32 mode, and with Number(n) under --number-mode f64, where the
message reads expected f64. Neither conversion works in the other mode. An array's slice is not a
slice panic site: its check is std/array.ts's own
panic(...), so a call to it is a call site that reaches panic, which no
proof at the call removes, and --deny-panics refuses it
(reject_deny_panics_arr_slice). Under Node and --emit-ts the call
is the runtime's sliceArray, which applies the same check with the same
words, so the two agree on every input; map, filter, some, every,
findIndex and forEach are the platform's own Array.prototype methods
there, and an answer that holds records inline is copied with the runtime's
copyRecords, as the native push copies.
Bulk writes: set and fill
The reference for set and fill, with their parameters and their answers, is their doc comments on Array in runtime/nish.d.ts; the rules are here.
A packet is copied, cleared and filled through u8[] without a loop per byte
(WP34 N2). Both methods take their meaning from TypedArray.prototype, and both
are admitted on an array of numbers only — u8[] above all, but any width,
f64[] and a ranged integer<Lo, Hi>[] included — because an element there is
a fixed-size value, so the lowering is one copy or one store of bytes. On any
other array each is `set` copies bytes, so it needs an array whose elements are numbers (`u8[]`, `i32[]`, `f64[]`, ...), got string[] (NL2395,
reject_bytes_set_string, reject_bytes_fill_record): a string or a class
element is a pointer a byte copy would share, and a record is a struct stored
whole.
-
a.set(src, offset)copies all ofsrcintoastarting atoffset, which defaults to0.srchasa's own element type — areadonlyarray may be the source, since it is only read — and anything else is`set` copies from an array of the receiver's element type, so it expects u8[], got i32[](NL2396,reject_bytes_set_type), because a conversion per element is not a byte copy. One or two arguments (NL2397,reject_bytes_set_arity). -
The range written is checked.
0 <= offsetandoffset + src.length <= a.length, or the program panics withslice out of range: [offset, offset + src.length) of length a.lengthand exits 1 — the same check and the same message ass.slice, since it asks the same question of a range (bytes_set_oob). A range that ends exactly ata.length, and an emptysrcata.length, are in range (bytes_set_end). JavaScript throws aRangeErrorfor exactly these offsets. -
An overlapping copy behaves as if
srcwere read first, which isTypedArray.prototype.set's promise, sosetlowers tollvm.memmove. Nish itself can overlap an array only with itself (a.set(a)), but a C host may hand in two headers over one buffer, and then the copy is right in either direction (bytes_set_overlap). It lowers tollvm.memcpyonly where the checker proves the two buffers distinct, and records the proof for the emitter: when either operand is an array literal or anewwritten at the call, or when both are differentconstlocals each initialised with a fresh allocation — a literal, anewor areadFileBytesSync— since a second name for either buffer would not be such a local (bytes_set_disjoint). A parameter, a field, aletand a self-copy are "not proven", which is thememmoveand never wrong (bytes_set_self). -
a.fill(value, start, end)storesvalueinto every slot of[start, end).valueis the element type (`fill` expects u8 (the element type of u8[]), got string,reject_bytes_fill_value), and a ranged element checks it once, before the fill.startdefaults to0andendtoa.length; a negative end counts back froma.length, and each is then clamped into[0, a.length], so no range is out of range, a reversed range fills nothing, andfillnever panics (bytes_fill). One to three arguments (NL2398,reject_bytes_fill_arity), each end a number (reject_bytes_fill_index). -
A float offset is converted as JavaScript's
ToIntegerOrInfinityconverts it, forset's offset and for both offill's ends: anf64(everynumberin f64 mode) orf32goes throughllvm.fptosi.sat, never a barefptosi, whose answer for NaN or an infinity is poison. NaN is0, a fraction truncates toward zero (2.7is2,-0.5is0), and an infinity or a value past thei64range saturates to thei64bound on its side.fillthen clamps that bound to an end of the array, so in f64 modea.fill(v, 0, 1 / 0)fills all of it, asUint8Array.prototype.filldoes (bytes_offset_float);Infinityitself is not a name in Nish.setfails its range check with the saturated offset in the message —slice out of range: [9223372036854775807, -9223372036854775808) of length 4for1e300or an infinity, where the end has wrapped — and exits 1 where JavaScript throws aRangeError(bytes_set_float_oob,bytes_set_float_panic). -
An unsigned offset is never negative, whatever its width. A
u64of 2^63 or more has thei64sign bit set, andfillstill reads it as past the end: as a start it fills nothing, and as an end it fills to the length (llvm.umin, where a signed clamp would count it back from the end).set's range check compares unsigned, so the same offset panics, printed signed as every range panic prints (bytes_offset_u64,bytes_set_u64_oob). Ani64atINT64_MINorINT64_MAXkeeps the signed clamp above. The TypeScript reading of a 64-bit offset is class C:i64andu64arebigintunderruntime/nish.mjs, andfillorsetwith abigintoffset throws aTypeErrorthere, as all 64-bit arithmetic does (RUN_UNDER_NODE.md). -
fillon au8[]is onellvm.memset. On a wider element it is a counted store loop of the one value, the shape LLVM vectorises;opt -O2's loop-idiom pass makes amemsetof that loop by itself when the value is a repeated byte, zero above all, whichtests/run.jschecks onbytes_fill_zero. -
Both are statements, as
writeFileSyncis:`set` returns void and can only be used as a statement(reject_bytes_set_value,reject_bytes_fill_value_position).setanswersundefinedin JavaScript andfillanswers the array the program already holds, so refusing the value costs nothing and keeps the two readings the same. -
Both write their receiver, so a
readonlyarray refuses them under their own names (Cannot `set` through readonly u8[] (declare it u8[] to write through it),reject_bytes_set_readonly,reject_bytes_fill_readonly), a scope's region refuses them on memory a task may read (reject_thread_region_fill,reject_thread_region_set_call), and a parameter they write is neverreadonly. Neither moveslength, so neither disturbs a hoisted header or a length the bounds proof holds. -
A window into a buffer is
(buf, offset, length)by convention, not a view type: a second array layout would be one every array path in the emitter has to handle (wp34-network-stack.md §4 N2).
The TypeScript reading (wp33-round-trip.md §2): fill is
class A — a u8[] is a plain Array under Node, and Array.prototype.fill
has TypedArray.prototype.fill's meaning for every argument Nish admits. set
is class A under runtime/nish.mjs, which adds the one method Array lacks
to Array.prototype with the copy-first semantics and the native panic, and
class C in a TypeScript host without it, where a plain array has no set
(wp33-round-trip.md §3.3). Every bytes_* program with
a main runs under both and agrees byte for byte.
Readonly arrays
The reference for the members a readonly T[] has is their doc comments on ReadonlyArray in runtime/nish.d.ts; the rules are here.
readonly T[] — or ReadonlyArray<T>, the same type under the other spelling
— is a T[] that may not be written through. It is the same header, the same
pointer and the same LLVM type; the golden
(tests/cases/arr_readonly_param) is the IR the mutable spelling emits,
instruction for instruction, because the annotation is a promise to the checker
and not a value.
-
Everything that reads still works:
a.length,a[i],for...of,indexOf,join,map,filter,some,every,findIndex,forEach,slice, passing it on to anotherreadonly T[]. -
Everything that writes is refused:
a[i] = vanda[i] op= vareCannot assign to an element of readonly i32[] (declare it i32[] to write through it)(reject_arr_readonly_store), andpush/popareCannot `push` through readonly i32[] (...)under their own names (reject_arr_readonly_push,reject_arr_readonly_pop), and so aresetandfill(reject_bytes_set_readonly).a.length = nwas already refused for every array. -
A
T[]widens into areadonly T[], and never back. The conversion emits nothing, because the two are the same pointer; the reverse isArgument 1 of `fill`: expected i32[], got readonly i32[](reject_arr_readonly_widen), since laundering the promise away one call deeper would make it worth nothing. -
It is shallow, as TypeScript's is: the element of a
readonly T[][]is a mutableT[]. -
It is a type, not a parameter modifier, so it is legal wherever an array
type is — a class field, a return type, a
const's annotation — and the rule travels with it (arr_readonly_field,reject_arr_readonly_field). That is why the message says "declare it" rather than naming a parameter. -
readonlygoes on an array type and nothing else. TypeScript allows the modifier on array and tuple types only (TS1354), soreadonly i32is`readonly` is only permitted on an array type, got i32(reject_arr_readonly_scalar) rather than a construct waiting to be supported — a program that used it would not be TypeScript either.
The reason to write one is the C ABI. --emit-header already spells an array
parameter const nish_array * when the whole-program fixpoint proves nothing
stores through it, which makes const a consequence that can quietly
disappear when a callee three levels down starts writing. On a readonly T[]
it is the signature keeping a promise instead, and the comment above the
prototype shows the annotation that earned it
(tests/cases/arr_readonly_header, checked in the WP8 block of
tests/run.js):
/* sum(xs: readonly number[]): number -- xs: int32_t elements */
int32_t sum(const nish_array *xs);
/* fill(xs: number[], v: number): void -- xs: int32_t elements */
void fill(nish_array *xs, int32_t v);
The fixpoint is not consulted for a readonly T[] at all, and that is
deliberate rather than a shortcut. writesThrough is a conservative
may-write: every escape sets it, on the grounds that an alias might be
written through later, so a readonly parameter that is merely returned or
stored in a field would land in the written set and lose the const its
annotation promised (tests/cases/arr_readonly_escape). The checker is what
makes the promise true — no store, no push, no pop, and no widening back to
a mutable T[] — and it is exact where the fixpoint is not
(writtenArrayParams, src/interop-abi.ts).
process.argv is read-only under a rule of its own rather than this type
(`process.argv` is read-only), because it is a value and not an
annotation.
There are no other array or string methods (shift, find, reduce,
toUpperCase, charAt, ...) and no
toString (template literals and console.log convert numbers);
string-to-number parsing is the bare
Number(s) / parseInt(s) / parseFloat(s) under
Numeric conversions. An unknown name is
Unknown method `codePointAt` on string or
Unknown method `shift` on i32[], each with the supported list
(tests/cases/reject_str_method_unknown, reject_arr_method_unknown).
Typed-array names have no push or pop
The reference for the four names is the doc comment on Array in runtime/nish.d.ts; the rules are here.
Int32Array, Float32Array, Float64Array and BigInt64Array are the same
type as i32[], f32[], f64[] and i64[], but in TypeScript a typed array
has a fixed length and no push or pop, so a call to either is a
TypeError there. A receiver spelled with one of the four names refuses
both (NL2415):
`push` is not a method of `Float64Array`, whose length is fixed: declare it `f64[]` to push and pop
(tests/cases/reject_typed_push_i32, reject_typed_push_f32,
reject_typed_push_f64, reject_typed_push_i64 and their reject_typed_pop_*
twins). The rewrite is the one the message names: a binding that grows is
declared f64[], which is what it meant. Every other array operation is
unchanged, and so is the type: sameType still holds, a Float64Array is an
f64[] wherever one is expected, and no IR moves.
The spelling is a fact about the program text, not a type. A receiver carries it when it is:
new X(n)itself, for the four names;-
an identifier whose
let,constor parameter is annotated with one of the names — directly, in parentheses, as the one member ofX | null, or through atypealias declared or imported in the module (reject_typed_push_param) — or is unannotated and initialised with a receiver that carries it:const t = new Float64Array(n), which TypeScript infers as the typed array (reject_typed_push_inferred), orconst u = t; - a field read
o.fwhose field is declared with one of the names; -
a call whose callee declares one of the names as its return type
(
reject_typed_pop_receivershas one of each). A field or a return type written through atypealias carries it in every module that reads it, including one that does not import the alias, because the declaring module reads the spelling where its aliases are (tests/link/reject_typed_push_alias); -
a call of a generic function whose return type is a type parameter —
or an array of one — that an argument binds to a receiver carrying it:
first(rows)withfirst<T>(xs: T[]): Tandrows: Float64Array[],same(new Int32Array(n))withsame<T>(x: T): T. The spelling is the call's, not the instantiation's:first(plain)over anf64[][]is the same instantiation and is not refused (reject_typed_push_generic); -
a value read out of a
Mapwhose value type is spelled with a name, whether theMapis annotatedMap<string, Float64Array>, made withnew Map<string, Float64Array>()or is a field so declared:m.get(k)once narrowed,m.get(k) ?? d, and the variable of afor...ofoverm.values()(reject_typed_push_map). ASetholds no arrays and aMap's keys are none, since an array is not a key; -
an element of an array spelled with a name:
rows[i]or afor...ofvariable overrows, whererowsis any of the above spelledFloat64Array[]orArray<Float64Array>, or is an array literal of receivers that carry it,[new Float64Array(n)](reject_typed_push_element). The array itself is an ordinary array, androws.push(new Float64Array(n))compiles; -
a ternary or a
??either of whose operands carries it:c ? xs : tisnumber[] | Float64Arrayto TypeScript, which has nopopeither (reject_typed_push_ternary).
What does not carry it is a member of a generic class instantiated at
one of the names: new Box<Float64Array>(t).value and a method of it that
returns T still push natively where the same source throws a TypeError
under Node, because a binding records one spelling and not one per type
argument. Declare such a value f64[] where it is bound, which is what
TypeScript needs to accept the push as well.
A typed array that flows into a binding spelled T[] may be pushed there:
const a: f64[] = t; a.push(v), or a Float64Array passed to a parameter
xs: f64[] that pushes (tests/cases/arr_typed_push_pop, whose IR is the
f64[] program's). The refusal follows the spelling the program wrote, and
TypeScript itself refuses const a: number[] = new Float64Array(n), so no
program tsc accepts reaches that push.
Arena
The reference for Arena.used, Arena.mark and arena(), with their answers and their capabilities, is their doc comments in runtime/nish.d.ts; the rules are here.
Reading the arena the runtime allocates from
(wp6-memory.md). The automatic placement
described under Memory model frees what a program allocates;
using a = arena() below frees a batch by hand, with the
proof done by the compiler. These two read the arena's position, and the
memory tests watch it through them. Both are available in both number modes.
Arena.release and Arena.reset, which freed to a mark with no proof, were
removed before 1.0: a call to either is an unknown builtin (NL2244, the
arena_removed block of tests/run.js).
| Signature | Semantics | Effect | Test |
|---|---|---|---|
Arena.mark(): i64 |
the current bump address (nish_arena_mark; 0 while the arena is empty) |
write | mem_callee_scope_control, mem_reclaim_guards |
Arena.used(): i64 |
bytes bumped in the current chunk (nish_arena_used); the number the memory tests watch |
write | mem_loop_scope_control, mem_scope_dynamic_array |
mark and used only read the arena, but they are recorded as writes so
that a caller is never hoisted across an allocation, and a function that calls
either — itself or through a callee — is never the callee of a tail call whose
scope release moved ahead of it, so the position it reports is the one it would
have reported before (Memory model, item 2). A function that
calls Arena.mark in its own body (managesArena) never gets an automatic
arena scope, for the function or for a loop's pass, so the position it reads
is never moved under it by a scope it did not write.
using a = arena()
A block that declares using a = arena() takes the arena's mark where the
declaration stands and releases to it on every exit of the block, and the
compiler refuses the block if anything allocated after the declaration could
still be reachable once it has released. It is the
per-pass scope of item 2 written by the program, held to the
same rule with the block as the pass, and refused rather than quietly dropped
when the rule fails.
const tally = (rows: string[]): i32 => {
let total = 0;
for (const row of rows) {
using a = arena(); // the mark
const line = `${row}: ${row.length}`;
total = total + line.length; // a number leaves the block; the string does not
} // released here, and on every other exit
return total;
};
-
arena()is a global builtin declared inruntime/nish.d.tsasdeclare function arena(): Disposable, so a program stays TypeScript; under Node its disposal does nothing. It is only the initialiser of ausingdeclaration, which is a statement of a block (theusingplacement rule); anywhere else it is`arena()` must be the initialiser of a `using` declaration, `using a = arena()`: the arena releases what was allocated after it when the block that declares it ends, and an `arena()` written anywhere else has no block to end(NL2416,tests/cases/reject_using_arena_not_using). The binding holds the mark and is never read:`a` is the binding of a `using` declaration of `arena()` and cannot be read: it holds the mark its block releases to when it ends, and nothing but that release may see or move the mark(NL2417,reject_using_arena_binding). -
The release is emitted at every exit of the block: its end, a
return(after the returned value is computed, so the value may read what the block allocated), abreakorcontinuethat leaves it, and anorReturn()(tests/link/arena_using_exit_paths,tests/cases/mem_using_arena,mem_using_arena_or_return). It nests with everything else on the same stack, innermost first: ausing s = scope()inside the block joins before the block releases, and a block inside a scoped loop pass or a function's automatic scope releases before they do; areturnreleases only the outermost of them, whose mark is the lowest. Areturn g(n)whose arguments are all numbers is a tail call with the release moved ahead of it, unlessgreads the arena (mem_using_arena_tail). A panic ends the process. Statements of the block before the declaration are outside it. -
Nothing allocated inside may outlive the block. From the declaration to the end of the block, each of these is refused, naming the value and where it would go:
Route Refused when Code Test assigned to a local declared outside the block the value is not older than the block (a literal, a parameter or outer local, a field or element read, a call that allocates nothing) NL2418 reject_using_arena_outer_localstored into a field, o.f = vvis not older than the block, unless the field is a number, abooleanor anenumNL2419 reject_using_arena_fieldstored into an element, xs[i] = vlikewise NL2420 reject_using_arena_elementstored into memory any other way: handed to a callee that keeps it, held by an object or array literal always: the escape analysis stops following a stored value NL2421 reject_using_arena_storedpushonto an arraythe array is not a constthe block binds to a fresh array, whatever is pushed: apushmoves the array's storage into the arenaNL2422 reject_using_arena_pushreturn v,r.orReturn()vis not older than the block;orReturn()unless the function returns aResultpacked into a register, whose payloads are numbersNL2423 reject_using_arena_return,reject_using_arena_or_return,reject_using_arena_declarator_or_returna call to a function that stores an allocation into memory always NL2424 reject_using_arena_callee_storesa call the analysis has no facts for always; every callee has facts today NL2425 — s.spawn(...)on a scope declared outside the blockalways: the task runs, and stores, when that scope joins, after the release NL2427 reject_using_arena_spawns.spawn(...)on a scope the sameusingdeclares ahead ofarena(),using s = scope(), a = arena()an argument is not older than the block: sis disposed after the arena, so it joins after the releaseNL2427 reject_using_arena_spawn_declaratorthe same, ahead of the statement's last arena(),using a = arena(), s = scope(), b = arena()an argument is not older than the block: breleases beforesjoinsNL2427 reject_using_arena_spawn_lastA number, a
booleanor anenumleaves the block freely, and so does a value older than it. The rule is the escape analysis's, so it is conservative: a block that stores a fresh string into an object it also allocated, or builds aMap, is refused even though nothing escapes, and what has to outlive the block is allocated before it opens. The first route reads`` `item ${i}` `` may be allocated inside the `using arena()` block at 6:5 and is assigned to `last`, which is declared outside the block: the block releases what it allocated when it ends, and the local would outlive it pointing into freed memory(reject_using_arena_outer_local). -
A parallel body or a task may not open one, for the reason it may not call
Arena.*(reject_using_arena_parallel, NL2351). -
A map that answers numbers or strings may run inside one. The map's region joins before the call returns, so its result array is the block's alone, and the blocks of strings it adopts from its workers are the block's newest memory; the release frees both (
tests/link/mem_using_arena_map). What the map answers may not outlive the block, as anything else allocated in it may not (tests/cases/reject_using_arena_map_escapes). A map whose body answers an array stores each one into its result, which the analysis stops following, as it does for a loop that pushes arrays into an array, so the block refuses it as NL2424 and says what to write instead: run that map outside the block (tests/cases/reject_using_arena_map_arrays). -
A task filed on a scope the block declares may read what it allocated. That scope joins before the block releases, inner
using s = scope()orusing a = arena(), s = scope()alike, so the block may hand its task an array or an object it made (tests/link/thread_scope_reclaim). A scope declared ahead of the statement's lastarena()is not one: that arena releases first (reject_using_arena_spawn_last). The task is followed as any callee is: one that answers its argument into an older destination stores it (NL2424,tests/cases/reject_using_arena_spawn_relay, asreject_using_arena_spawn_storesfor any array answer), and a constructor that makes the argument and handsthisout stores it too (reject_using_arena_spawn_constructor). -
It costs no scope. A function that declares one keeps its automatic scope and its loops' pass scopes, because the block's mark is checked like theirs; only
Arena.markgives those up. And the arena-loop warning, NL9011, says nothing about a call inside the block, whose leftovers the block releases (tests/cases/perf_arena_using).
Map and Set
The reference for each member of Map and Set, with its parameters and its answer, is its doc comment in runtime/nish.d.ts; the rules are here.
The global Map<K, V> and Set<T> are JavaScript's: insertion-ordered, with
keys compared by SameValueZero, so -0 is +0 and NaN is NaN. They need
no import. They are generic classes written in Nish in
std/collections.ts, which a module loads by naming
Map or Set in a type or after new, and the layout and every decision
below are wp32-map.md's.
export const main = (): i32 => {
const seen = new Set<string>();
const counts: Map<string, i32> = new Map();
seen.add("a").add("b").add("a");
counts.set("a", 1).set("b", 2);
counts.delete("b");
return seen.size + counts.size; // 3
};
-
The members are JavaScript's, less what this version defers. A
Maphassize,get,set,has,delete,clearandforEach; aSethassize,add,has,delete,clearandforEach.setandaddanswer the receiver, so they chain, anddeleteanswers whether the key was there (tests/cases/map_key_str,set_str).sizeis a read-onlynumber:`size` of `Map<string, i32>` is read-only(reject_map_size_assign).getanswersV | undefined, which has rules of its own, below.keys()andvalues()are iterators, and an iterator is only ever the iterable of afor...of, below. There is noentries, because a[key, value]pair is a two-type tuple (reject_map_entries,reject_map_iter_entries_call):m.forEach((value, key) => ...)visits each key with its value. -
forEachis thefor...ofwalk with the callback inlined.m.forEach(f)on aMapcallsfwith each value, and with its key too whenfis written with two parameters, value first, as JavaScript does;s.forEach(f)on aSetcallsfwith each element. The checker rewrites the call to a template ofstd/collections.ts(forEachEntry,forEachValue,forEachElement), whose body is the same walk asfor...ofover the table, below: the same insertion order, an entry set during the walk visited and one deleted before the walk reaches it not, andclearending it (map_foreach,set_foreach). It allocates nothing and looks no key up, wherefor (const k of m.keys())withm.get(k) ?? 0callsMap.probeand takes the nullish branch once per key, and rewinds the arena on every pass: underopt -O2, 92 instructions against 112 for aMapand 78 against 89 for aSet(IR_COOKBOOK.md). JavaScript passes aMap's callback the map as a third argument, and aSet's callback the element twice and the set third; Nish refuses a callback written with more than two parameters on aMap(NL2771,reject_map_foreach_third_param) or more than one on aSet(NL2772,reject_set_foreach_second_param), since the receiver is the one the call already names. The callback follows the array callbacks' rules above: a top-level function or an arrow written at the call (reject_map_foreach), at least one parameter, an answer ofvoid, no assignment to a name of the function around it (NL2586), andm.forEachnamed without a call is refused (reject_map_foreach_uncalled). An arrow inside another arrow reads only its own parameters, so a nested walk goes through a top-level function (set_foreach). Everything else in the class is the table's own and is refused as if it did not exist:Unknown member `mask` on `Map<string, i32>`(reject_map_internal_field,reject_map_internal_method,reject_set_set). -
newtakes no arguments, because a class of this language has one constructor and no optional parameter:`new Map<string, i32>` takes no arguments in this version(reject_map_new_args,reject_set_new_args). The type arguments are written out, except that a declaration annotated with the type takes them from the annotation, astscinfers them there —const m: Map<string, i32> = new Map()and the same withlet(tests/cases/map_annotated_new); a barenew Map()is`Map` is generic: it must be written with its type arguments(reject_map_new_no_type_args). -
A key is a string, a number of any width in either mode,
f32,f64, aboolean, an enum or a class instance, the last by identity (tests/cases/map_key_str,map_key_i32,map_key_i64,map_key_u8,map_key_u16,map_key_u32,map_key_u64,map_key_f64,map_key_f32,map_key_bool,map_key_enum,map_key_class). An interface, an array, a nullable type and aResultare not keys:`Point` cannot be the key type of `Map<Point, i32>`: a key is hashed and compared by value, or by identity for a class instance(reject_map_key_interface,reject_map_key_array,reject_map_key_nullable,reject_map_key_result, and for aSetelement,reject_set_key_interface). A float key is SameValueZero, and so is its hash:-0and+0are one key and everyNaNis one key, andmap_key_f64prints under Node's ownMapwhat it prints compiled (tests/differential/unmodified.js). A-0key is stored as+0, as JavaScript stores it, so a walk overkeys()or aSetyields+0whichever zero was inserted, by every insert path:set,add, a fused update or guarded insert, andgetOrInsert(map_key_negzero, which Node runs unmodified too, andmap_key_f32_negzerofor anf32key). -
A value is anything but
voidand an interface, whose record would be copied into the table rather than shared:`Point` cannot be the value type of `Map<string, Point>`(reject_map_value_interface,reject_map_value_void). -
A string key is hashed under a per-process random key: SipHash-1-3 of its UTF-8 bytes, keyed with 128 bits drawn from the system's entropy source before
main, so a program's input cannot choose keys that share a bucket. 4096 keys that collide under an unkeyed FNV-1a insert with no key compare (tests/cases/map_hash_collide, counted the same way as below), and two runs of one program hash under two keys. Iteration is still insertion order, so nothing a program prints depends on the key. When the entropy source fails the program panics beforemain:Map: the system's entropy source failed. On wasm32 the key comes from WASI'srandom_get, and a reactor module is keyed only once its host calls_initialize(docs/security/json.md, JSON-13). -
An integer,
booleanor enum key is hashed under a per-process random key too, drawn the same way: a key of 32 bits or fewer by multiply-add-shift,(a * x + b) >> 32, and ani64oru64key by a folded 128-bit multiply. 8192i32keys built to collide under 0.28's unkeyedfmix32run within 8 times of as many ordinary keys, where 0.28 ran them about 215 times slower (tests/cases/map_key_int_collide). Thei64hash is keyed but not provably universal, and before_initializea wasm32 reactor hashes every key of 32 bits or fewer to one bucket (docs/security/json.md, JSON-15). A float key and a class instance key are not keyed: a float hashes its canonical bits withfmix64and an instance its address, so a program keying a table by floats a peer chose converts them to a string or a scaled integer first (JSON-16). -
A pair of
i32s is onei64key, packed exactly for every pair:(toI64(hi) << toI64(32)) | toI64(toU32(lo)).toU32takeslounsigned, so it stays in the low half, so nothing is added and nothing overflows; with+in place of|the key leaves thei64range athi = -2^31with a negativelo. Every operand is ani64so the same line runs under Node, where ani64is aBigInt(tests/cases/map_key_composite, both extremes included). -
Every
set,add,hasanddeleteis one probe. A bucket holds eight bits of the key's hash beside the entry's index, and each entry keeps its full hash, so a probe reads a key only when both match: 10000 lookups of absent string keys make no key compare at all, and 10000 hits make exactly 10000 (tests/cases/map_fingerprint_miss, counted bytests/run.jsthrough-Wl,--wrap=nish_str_eq). A table grows by doubling at three quarters full and never hashes a key again (map_growth); a deleted key leaves a tombstone that later probes walk past, and setting it again appends a new entry (map_delete_reinsert); and a table whose entries are mostly dead compacts in place instead of growing (map_compaction). A table holds at most 2^24 - 1 entries, which is what a 24-bit index field holds and one fewer than Node's 2^24: an insert at the cap first compacts away dead entries, and when there are none it panics withMap maximum size exceeded(orSet …) (CLI only: the case takes 16 million inserts). -
Asking about one key twice is one probe (wp32-map.md §9.1). Three patterns keep the first probe's answer — the entry it found, or the empty bucket it stopped at and the key's hash — and write through it:
setValueAtof the entry where the key was found, andinsertAtthe bucket with the hash already computed where it was not. The second hash and the second search are never made:Pattern Written Case update counts.set(w, (counts.get(w) ?? 0) + 1): the value asks about the key exactly once, withgetorhasmap_fused_wordcount,map_fused_update_hasguarded write if (m.has(k)) { m.set(k, E); … }orif (!m.has(k)) { m.set(k, E); … }, thesetfirst in the branchmap_fused_guard_has,map_fused_guard_not_hasinsert if absent if (!s.has(x)) { s.add(x); … }, theaddfirst in the branch; the rest of the branch runs only where it insertedmap_fused_set_insertThe receiver and the key are spelled the same way in both calls, and each is a local, a parameter or a
this.<field>path; the key may also be a literal (map_fused_this_field,map_fused_literal_key). Between the probe and the write — inside the value, and between thehasand the branch's first statement — there is no call of any kind, nonew, array or object literal, template literal or string+, and no assignment,++,--or compound assignment;&&,||,??and the ternary may stay, and a??default still runs only where the key is missing. A pattern outside these rules is not an error: it compiles as the separate calls it is, one probe each, and prints what they print (map_fused_no_call,map_fused_no_assign,map_fused_no_key_spelling,map_fused_no_element_receiver,map_fused_no_template,map_fused_no_statement_before). The places are compared, not what they hold, so an alias of the receiver is not fused (map_fused_no_alias), and neither is a value that reads the key twice or reads another map (map_fused_no_two_gets,map_fused_no_other_map); a guard fuses the write that starts its then-branch, and asetin itselseis an ordinary call (map_fused_no_else_branch). A pattern in a generic function or method is fused in every instantiation (map_fused_generic), and one inside a walk of the table it writes does what the separate calls do: a key it inserts is appended past the cursor and visited (map_fused_walk).tests/run.jscounts one call ofprobe, no hash of its own and noget,setorhasin the loop ofmap_fused_wordcount's golden (IR_COOKBOOK.md). -
nish/mapaddsreserve(m, n)andgetOrInsert(m, k, v), the two things a program asks of a table that JavaScript'sMaphas no call for (wp32-map.md §9.2). Their bodies instd/map.tsare what runs under Node —reservedoes nothing, andgetOrInsertis aget, and asetofvwhen the key was missing, answering the value either way — so a program that uses them runs unmodified there (examples/wordcount.ts). Natively neither body is called: every call is lowered in place, sostd/map.tswrites no.llof its own and a one-file program importing it is still one module.import { getOrInsert, reserve } from "nish/map"; export const main = (): i32 => { const first = new Map<string, i32>(); reserve(first, 64); // no rebuild until 64 entries const a = getOrInsert(first, "a", 1); // 1, inserted const b = getOrInsert(first, "a", 2); // 1, found: 2 is not stored return a + b + first.size; // 3 };-
getOrInsertis one probe: the value of the entry found, orinsertAtofvthrough the bucket the probe stopped at.vis evaluated either way, as every argument is (tests/cases/map_extras_get_or_insert, whose looptests/run.jscounts as oneprobeand nogetOrInsertcall). -
reserveis the table'sreserveSlots, which doubles the bucket table untilnentries, dead ones included, fit under the three-quarters bound, and re-files the buckets from the stored hashes. It never shrinks a table, a count that is not positive does nothing, and one past the 2^24 - 1 entry cap is the cap (map_extras_reserve). It moves no entry, so likegetOrInsertit may be called during a walk of the table (map_extras_walk), and both are lowered in each instantiation of a generic caller (map_extras_generic) and in each module that calls them (tests/link/map_extras_two_modules). Only speed depends on it: each case prints under Node what it prints compiled (tests/differential/unmodified.js). -
They are generic functions, checked as one: a first argument that is
not a
Map, or a key or value of the wrong type, isArgument 1 of `getOrInsert<i32, i32>`: expected Map<i32, i32>, got i32[](reject_map_extras_not_map,reject_map_extras_key_type,reject_map_extras_value_type, andreject_map_extras_reserve_countfor a count that is not anumber); a missing argument is`getOrInsert` expects 3 argument(s), got 2(reject_map_extras_arity); and aSetis not a tablereservetakes:Cannot infer `K` for `reserve`: argument 1 is Set<string>, which does not match the declared `Map<K, V>`(reject_map_extras_reserve_set). The module has those two and nothing else:Module `nish/map` has no exported function `peek`(reject_map_extras_unknown_export, NL2376, the code any module's missing export now carries). -
The lowering belongs to
nish/map's two functions, found by the module that declares them and not by their names: a program's ownreserve, or agetOrInsertfrom its own module./map, is an ordinary function whose body runs (tests/link/map_extras_user_names).
-
-
std/collections.tswrites no.llof its own. What a module uses of it is emitted into that module,internal, after the module's own functions, so a one-file program that namesMapis still one module for-o <file.ll>and every instance is inlined, specialised or dropped with the rest of the module. Two modules that share oneMap<string, i32>each call their own copy of its methods (tests/link/map_two_modules). Under-geach copy's subprogram namesstd/collections.ts(tests/cases/map_dbg). The symbols are the package's (@nish.Map$str$i32.set), and a map is filled and grown through a callee and across loop arena scopes like any other object (tests/cases/map_arena_callee). -
m.get(k)answersV | undefined, as it does undertsc: a maybe. It is one probe, and a maybe is read in one of three ways, the three a program that type-checks undertsc --stricthas (wp32-map.md §3):export const main = (): i32 => { const counts = new Map<string, i32>(); for (const w of ["a", "b", "a"]) { counts.set(w, (counts.get(w) ?? 0) + 1); // a default } const a = counts.get("a"); // a `const`, tested if (a !== undefined) { console.log(`${a}`); // 2: `a` is an i32 here } return counts.get("z") === undefined ? 0 : 1; // a test };-
Where a maybe may stand: the initialiser of a
const, unannotated or annotated with the maybe type; the left operand of??; and an operand of=== undefinedor!== undefined, either way round (tests/cases/map_get_nullish,map_get_narrow). The type is spelledV | undefinedorundefined | V, andNode | null | undefinedfor a nullableV(a scalar is no more nullable here than anywhere,reject_map_get_nullable_scalar), and only as the annotation of aconstinitialised from a call ofget. On aletinitialised fromgetit is refused as the unannotatedletis (NL2361,reject_map_get_let_annotated), and anywhere else it is the union it looks like (reject_union_undefined), andundefinedas a value is refused as it always was (reject_undefined_value, and in a module constant, which never holds agetresult,reject_map_get_undefined_module_const). -
A
constbound togetis narrowed by the rules of Nullable types, withundefinedin place ofnull: it reads asVin the region a test proves it present,if (a === undefined) returnincluded, and nowhere else (map_get_narrow,reject_map_get_narrowing_leaks). Aconstis never assigned, so its narrowing never ends early; aletwould need that rule, and is refused. Narrowed, it is still the maybe it was bound to, as undertsc: testing it again, or defaulting it with??, reads the same found bit (map_get_narrowed_again). Compared withundefined, anything that is not a maybe is`undefined` is forbidden in Nish; use `null` with a `T | null` type(reject_map_get_undefined_compare). -
a ?? dhas TypeScript's precedence: the level of||, with operands that bind as tightly as|, left associative (tests/parser/nullish). It does not mix with||or&&without parentheses, astscreports TS5076:`||` and `??` cannot be mixed without parenthesesis a syntax error (reject_nullish_mixed_or,reject_nullish_mixed_and), reported once however often an expression mixes them (reject_nullish_mixed_chain_or,reject_nullish_mixed_chain_and).druns only when the value is missing, and it has typeV, as does the whole expression:The right operand of `??` stands in for a missing value, so it must be `i32`(reject_nullish_default_type). WhenVis nullable,??replaces a storednullas well, as JavaScript's does, and the result is stillV;!== undefinednarrows toV, keeping thenull(map_get_pointer_value). A??whose left operand is aT | nullrather than a maybe is Nullable types' default, and one whose left operand is neither is refused in the words Phase 0 always used,Nullish coalescing `??` is forbidden in Nish (narrow with `!== null` instead)(reject_nullish,reject_nullish_nullable), which thenullishfeature of the compatibility dialect admits. -
Everywhere else a maybe is refused, and the message names the place, because the rewrite differs by place. Some fail under
tsctoo (an operand, aconstannotatedV); the rest are legal TypeScript that this version does not take, because a maybe never crosses a call, never reaches memory, and is narrowed only in aconst(wp32-map.md §3.2):Place Message after `m.get("a")` is `i32 | undefined`Case a let, or an assignment to oneand cannot be held in a `let`reject_map_get_let,reject_map_get_assign_let,reject_map_get_let_annotated(NL2361)an argument of a function, method, constructor or builtin and cannot be passed as an argumentreject_map_get_argument,_argument_method,_argument_constructor,_argument_builtin(NL2362)a return, or a concise arrow bodyand cannot be returnedreject_map_get_return,reject_map_get_const_read(NL2363)a field, or an object literal's and cannot be stored in a fieldreject_map_get_field,reject_map_get_object_field(NL2364)an array element, or an array literal's and cannot be stored in an array elementreject_map_get_element,reject_map_get_array_literal(NL2365)a template literal hole and cannot be a template literal holereject_map_get_template(NL2366)an operand of any other operator and cannot be the operand of an operator other thanreject_map_get_arithmetic,reject_map_get_compare,reject_map_get_unary(NL2367)the default of another ??(m.get(a) ?? m.get(b) ?? 0)and cannot be the default of another `??`reject_nullish_default_maybe(NL2371); writem.get(a) ?? (m.get(b) ?? 0)a constannotatedVand cannot initialise a `const` annotated with its value typereject_map_get_annotated(NL2368)anywhere else, a dropped result included , which can only initialise a `const`, be the left operand of `??`, or be compared with `undefined`reject_map_get_statement(NL2369)getis called, never read as a value (Unknown field `get` on class `Map<string, i32>` (it is a method; call it),reject_map_get_property), and its key is checked againstK(reject_map_get_key_type,reject_map_get_arity). -
A maybe is two SSA values: the found bit, which is the probe's packed answer compared with 0, and the value, which
valueAtreads only on the edge where the bit is set. There is no struct and no store. Eachgetin themap_get_*goldens is exactly one call of the instance'sprobe, andgetnever probes twice nor callshas.??on agetis a branch to the value or the default and aphi; on aconst, whose value is loaded already, with a literal or a name as its default, it is oneselect(map_get_narrow). Where the??is wanted as an interface its class implements, the two class pointers are joined and the result converted once (map_get_interface). Aconstholds aphiof the value andV's zero, which no narrowed read can reach. A value built in the default goes wherever the??goes, as a ternary's arm does, so an object made there and stored in the map is allocated in the arena (map_get_default_escapes). Each value width is a golden:map_get_value_i32,map_get_value_f64,map_get_value_bool,map_get_value_string, an enum,i64,u64andf32inmap_get_value_widths, and a class inmap_get_pointer_value. Under-gaconstbound togetis described byllvm.dbg.valueof its payload, since it has no slot (dbg_map_get). Thenumber-mode cases print under Node's ownMapwhat they print compiled (tests/differential/unmodified.js).
-
-
for...ofwalks a table in insertion order:for (const k of m.keys()),for (const v of m.values()), and for aSet,for (const x of s),s.keys()ands.values(), which are one walk, as JavaScript'sSet.prototype.keysisvalues(wp32-map.md §6.2, §6.3;tests/cases/map_iter_keys,map_iter_values,set_iter,set_iter_keys,set_iter_values). The variable takesKorV, and a deleted key is not visited. A walk in a generic function or method is checked per instantiation like the rest of its body (map_iter_generic).export const main = (): i32 => { const ages = new Map<string, i32>(); ages.set("ada", 36).set("alan", 41).set("grace", 85); ages.delete("alan"); let total: i32 = 0; for (const name of ages.keys()) { console.log(name); // ada, then grace } for (const age of ages.values()) { total += age; } return total === 121 ? 0 : 1; };-
The table may change during the walk, with JavaScript's effect, because the walk re-reads the entry count every pass and no entry moves while a loop walks the table. Each row is a golden whose output Node's own
Mapprints too (tests/differential/unmodified.js):During the walk Effect Case setof a new keyvisited later in the same walk map_iter_set_new,set_iter_valuessetof a key already therethe new value is seen if the walk has not reached the key yet, and the key keeps its place map_iter_set_existingdeleteof a key not yet visitedit is skipped map_iter_delete_aheaddelete, thenset, of one keyit is visited again, at the end map_iter_delete_resetclearnothing more is visited, then whatever is set after it is map_iter_clearenough inserts to grow the table invisible: the buckets move, the entries do not map_iter_growth -
A walk defers compaction. Compacting is the one thing that moves entries, so each table counts the loops walking it: the count goes up where a loop is entered and down on every edge that leaves it — the fall-through,
break, areturnor anorReturn()out of any number of walks around it — and not oncontinue. A rebuild while it is above zero doubles the table instead, and the first rebuild after the walk compacts. Churn during a walk moves nothing under it (map_iter_compaction_deferred), and churn after a walk left by each kind of edge allocates nothing once warm, which is a compacting table (map_iter_break,map_iter_return,map_iter_or_return). Walks are lexical loops, so nested walks of one table and a walk in a function called from a walk are counted exactly (map_iter_nested,map_iter_callee). Abreakout of aswitchinside a walk leaves the switch and not the walk, areturnfrom inside it closes the walk, and a receiver reassigned mid-loop does not change the table the walk closes, which is the one it opened (map_iter_exit_edges). It costs two stores a loop and none an entry. A walk never counts as a bounded loop: a body that sets a new key every pass never ends, as in JavaScript. An insert that would pass the 2^24 - 1 entry cap while a loop walks the table panics rather than compacting. -
The walk is four calls of the table's own code:
walkOpenwhere the loop is entered,walkNextfor the first live entry and after each pass,keyAtorvalueAtfor the variable, andwalkCloseon each edge out, all emittedinternalinto each module that walks, like the rest of the table (tests/link/map_walk_two_modules) (IR_COOKBOOK.md). Under-gthe variable is described in its slot like anyfor...ofvariable (dbg_map_iter). -
An iterator is not a value, so
keys()andvalues()anywhere but as the iterable of afor...ofare refused, naming the place:Place Message after `keys()` of `Map<string, i32>`Case a const, alet, or an assignmentcannot be storedreject_map_iter_const,reject_map_iter_assign,reject_map_keys(NL2372)an argument cannot be passed as an argumentreject_map_iter_argument(NL2373)a return, or a concise arrow bodycannot be returnedreject_map_iter_returned(NL2374)anywhere else, a dropped result or an uncalled m.keysincludedcan only be the iterable of a `for...of`reject_map_iter_statement,reject_map_iter_uncalled,reject_map_values(NL2358)for (const e of m)over aMapitself would walkentries(), whose[key, value]pairs are two-type tuples:`for...of` over `Map<string, i32>` needs `entries()`, with the rewrite,for (const k of m.keys())(reject_map_iter_entries, NL2375). It reports once: the variable is an error that its body reads without a second report (reject_map_iter_recovery). A spread is refused as it is in any array literal (reject_map_iter_spread, NL2237),for...inis Phase 0's NL1056 as it is over anything (reject_map_iter_for_in),keys()takes no argument (reject_map_iter_arity), and the variable has the type it walks (reject_map_iter_variable_type).
-
-
A module's own
MaporSetwins in that module. A module that declares or imports a class, interface, alias or enum of either name gets no implicit import, as the declaration shadows the global undertsc(tests/link/map_own_class), and itsgetis its own method, answering what its signature says (tests/cases/map_get_user_class). A type parameter calledMaporSetis not a use of the global either, so a module whose onlyMaporSetis a type parameter loads nothing and compiles as it did beforeMapexisted (tests/cases/map_type_param_shadow). It does not shadow the global, though: as for any class name in scope, a module that also uses the globalMaprefuses a type parameter of that name with`Map` is generic: it must be written with its type arguments(tests/cases/reject_map_type_param_beside_global), wheretscwould shadow it. A class name is program-wide, though, so a program in which one module declaresMapand another names the global is refused at the declaration:`Map` is declared here and main.ts uses the global `Map`; a class or interface name is program-wide, so one program cannot have both(tests/link/map_clash, NL3030). -
Interop:
--emit-headerdeclares an instance as an opaque struct,typedef struct nish_gen_Map_str_i32 nish_gen_Map_str_i32;, so a C host can hold one it was given and pass it back;--emit-dtsand--emit-napileave a function that takes or returns one unbridged, with the comment they write for a class.
Typed JSON: nish/json-typed
The reference for each export, with its parameters, its answer and the codes it records, is its doc comment in std/json-typed.ts; the rules are here.
jsonDecode(d, out) reads one JSON object into a class and jsonEncode(w, value) writes one back, and the code that does it is written by the compiler
from the class's fields. They are generic functions in
std/json-typed.ts, and every call of one is lowered
in place, in the module that makes it, so nothing is looked up by name at run
time and no class carries anything it would not otherwise have.
import { JsonWriter } from "nish/json";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDecoder, jsonDecode, jsonEncode } from "nish/json-typed";
class Player {
name: string = "";
score: i32 = 0;
}
class Team {
players: Player[];
captain: Player | null = null;
constructor() {
this.players = [];
}
}
export const main = (): i32 => {
const d = new JsonDecoder(body, 0, toI32(body.length), new JsonLimits());
const t = new Team();
const code = jsonDecode(d, t); // 0, or the first error's code
const w = new JsonWriter(out, 0);
jsonEncode(w, t); // {"players":[...],"captain":...}
return code;
};
-
The decoder is
JsonDecoder, aJsonTokenizerover the input and a sticky error, and the encoder is theJsonWriterthe call is given. The code written for a class is straight-line calls of their methods, one per field: for decoding, a loop over the object's keys that sends each key to the field it names and reads the value with the method of the field's type; for encoding,w.key(name)and the value's method per field, in declaration order. A nested class, an array's element class, a| nullfield's class and aMap's values each get the same code of their own, so it recurses as the data does (tests/link/std_json_typed, whosemain.llis the golden of the code written for a one-field class andtag.llof the code for au64, an enum and arrays of each; IR_COOKBOOK.md). Because the encoding is calls of the writer's methods, it is written in the writer's modes: a writer set withsetIndent(2)encodes whatJSON.stringify(v, null, 2)writes, and asetStrict(true)writer fails a NaN or infinitef64field withJSON_WRITE_NON_FINITE(tests/link/std_json_writer). -
A key is sent to its field by its length, then by one byte. The code written for a class of three or more fields switches on the key's length in bytes, so a key is compared only with the names of that length. Where three or more names share a length, it switches once more, on the key's byte at the one position where those names differ most, and calls
keyIsfor each name that has that byte there, in declaration order; a byte none of them has is an unknown key at once. One or two names, of a class or of one length, are compared in turn withkeyIs, which is no more calls. A key with an escape has no byte to switch on there, and is compared with every name of its length, so"\u0069d"is the fieldid(tests/link/std_json_typed, "wide: every key escaped"). A key is therefore compared with the names of its length that share its byte at that position, usually one, rather than with every name in turn, and a field name that is not ASCII is dispatched by its UTF-8 bytes like any other ("names: escaped field names decode"). -
What a field may be.
Tis a declared class, and every field of every class the call reaches is one of these; anything else is refused at the call's argument, naming the class and the field:Field type JSON Decoded with Encoded with i32,u32,i64an integer written without a fraction or an exponent, in the type's range d.i32(),d.u32(),d.i64()w.i32,w.u32,w.i64u64an integer written in digits alone, with no sign, fraction or exponent, up to 2^64 - 1 d.u64(), through the tokenizer'snumberU64jsonEncodeU64a numeric enumEthe number of one of E's membersd.i32(), then aswitchover the members' valuesw.i32a string enumEthe string of one of E's membersd.enumString(), then the string's bytes against the members' stringsw.stringof the member's stringf64(andnumberunder--number-mode f64)any number; past the double range an infinity, which the writer writes as nulld.f64()w.f64booleantrue,falsed.boolean()w.booleanstring,string | nulla string, and nullfor the secondd.string(cur)w.string,jsonEncodeStringOrNulla declared class C,C | nullan object, and nullfor the secondC's own decoding, into the object the field holdsC's own encodingT[],T[] | nullfor any of the abovean array, and nullfor the secondd.i32Array(cur)and its kin, orC's orE's decoding per elementjsonEncodeI32Arrayand its kin, orC's orE's encoding per elementMap<string, V>,Map<string, V> | null, for anyVin this table,| nullor not, aMapincludedan object whose keys are the map's, each value a V, andnullfor the secondthe object's keys in order, each value decoded as a Vandsetthe map's entries in its order, each value encoded as a Va tagged union Uof declared classes,U | null,U[]an object whose tag key names its variant, and nullfor the secondthe tag's string, then that variant's decoding, into the object the field holds when its tag matches the tag first, then the variant's own encoding JsonNumberfromnish/json-typed, alone,| null, in an array or as aMap's valueany number, kept as the text it is written in d.number(n), which copies the number's bytes intonjsonEncodeNumber, which writes them back`jsonEncode` cannot handle `Sample.ratio`, which is `f32`: a field read or written as JSON is an `i32`, `u32`, `i64`, `f64`, `boolean`, `string` or a declared class, an array of one of those, or a `string`, a class or an array with `| null`(NL2485,tests/cases/json_reject_field_type; two classes down through an array's element,json_reject_nested_field; in a| nullfield's class,json_reject_nullable_field; in a class aMapholds,json_reject_map_value_class;i32[][],json_reject_nested_array, andLevel[][],json_reject_enum_nested_array; a generic class of the program's own,json_reject_generic_field). The message goes on to name the kinds added since:a `u64` and an enum are too, alone or in an array, and so is a `Map` whose keys are `string` and whose values are any of these, with `| null` or not. A JSON object's keys are strings, so aMapkeyed by anything else is refused, an enum included (json_reject_map_key,json_reject_map_key_enum), and so is a map whose values typed JSON cannot write, at any depth (json_reject_map_value,json_reject_map_value_nested). ASetand an array of maps are refused as well (json_reject_set_field,json_reject_map_array). AMapkeyed by a string enum is refused with the rest (json_reject_map_key_enum_string): a map's keys are read and written asstrings, and a string enum is not one. An enum that mixes strings and numbers is not one the language declares at all (NL1043,json_reject_enum_mixed), an enum is never| null, numeric or string, so aRole | nullfield is refused where it is written (NL2010,json_reject_enum_nullable) while aRole[] | nullis a field like any other, and aconst enumis refused where it is declared (NL2285,json_reject_const_enum). A value that is not a declared class is`jsonEncode`'s value is `Point`, which is not a class: typed JSON reads and writes an object through the fields of a declared class, and an interface, a generic class or any other type is not one in this version(NL2486,json_reject_interface,json_reject_not_class). An interface is a record stored inline wherever it is held, and decoding into one is left for a later version. A call inside a generic function is checked for each class an instantiation binds (json_reject_through_generic). -
Decoding is strict, and every outcome is a code. The keys are the field names, byte for byte, or the names the class's options give them. Every field is required unless its class's options make it optional, each key may appear once, and a key that names no field is an error unless
d.ignoreUnknownis set, when the key's value is stepped over, checked but not kept. The first error setsd.failedandd.errorAt, the byte offset of the token it was found at, and every later call does nothing, as the tokenizer's and the writer's errors do;jsonDecodeanswersd.failed, andd.resetstarts over:Outcome Code errorAta value of the wrong JSON type: a string for an i32, an array for a class,nullfor a field that is not| null, a top-level value that is not an object, an integer that is no member's value for a numeric enum, a string that is no member's value, or a value that is not a string, for a string enumJSON_DECODE_TYPE(-20)the value a number its field cannot hold: a fraction or an exponent for an integer, or out of range, a negative number ( -0included) for au64JSON_DECODE_RANGE(-21)the number a key that names no field JSON_DECODE_UNKNOWN(-22)the key a key given twice, a field's or a Map's, however it is escapedJSON_DECODE_DUPLICATE(-23)the second a field the object does not give JSON_DECODE_MISSING(-24)the object's }more keys, over every Mapfield of the decoding, thand.maxMapKeys(4,096 unless set)JSON_DECODE_MAP_LIMIT(-25)the first key past it a union's tag missing, not a string, or a string no variant writes JSON_DECODE_TAG(-26)the tag's value, or the object's {when it has nonetext that is not JSON, or that crosses a JsonLimitslimitthe tokenizer's JSON_TOK_ERR_*(-1 to -13)the tokenizer's Nothing an input holds panics: the recursion is the data's, so the tokenizer's depth limit bounds it. That bound is the caller's
JsonLimits.maxDepth, 128 by default: each level is one native frame, so a limit raised into the hundreds of thousands can run off the stack, and one kept to a few thousand cannot. The writer refuses a 65th open container (JSON_WRITER_MAX_DEPTH), so a value decoded from deeper than 64 levels encodes asJSON_WRITE_TOO_DEEP(tests/link/std_json_typed, which reaches each row at each depth, the limits included). A duplicate of a keyignoreUnknownsteps over is not detected, because its value is never read. After an error every field ofoutholds a value of its type, some decoded and some as they were: an enum field or element never holds a number that is no member's, because such a number is not stored. -
A numeric enum is its number. It is read with
d.i32(), so a number that is not ani32integer is that field'sJSON_DECODE_RANGE, and an integer that is not the value of one of its members isJSON_DECODE_TYPE; a member's name,"High", is a string and a type error too. Two members of one value are one number (tests/link/std_json_typed, "enum: two members of one value both read 1"). It is written asw.i32of its value, asJSON.stringifywrites a TypeScript numeric enum. -
A string enum is its member's string. It is read with
d.enumString(), which takes the value as a string without copying it, and its bytes are matched against the members' strings the way a key is sent to its field: by length, then by one byte, then withkeyIs, so the read allocates nothing, and an escaped string,"\u0061dmin", is the member its unescaped bytes spell. A string that is no member's, a prefix of one included, and a value that is not a string areJSON_DECODE_TYPE, and the field or element keeps the member it held. It is written asw.stringof the member's string, which is whatJSON.stringifywrites for a TypeScript string enum (tests/link/json_enum_string; IR_COOKBOOK.md). A field, an array, a| nullarray and aMap<string, Role>'s values are read and written so. -
A
u64is exact to 2^64 - 1. It is read through the tokenizer'snumberU64, so18446744073709551615decodes and18446744073709551616,-1,-0and1.5areJSON_DECODE_RANGE. It is written as its digits, whichJSON.stringifycannot do for a number past 2^53:tests/run.jsholdsround-trip-kinds.txttoJSON.stringify(JSON.parse(line))with each integer past 2^53 kept as its own digits (JSON.rawJSON). -
A class's options are one
static readonly jsonliteral. A class gives a field another key on the wire, and lets a key be left out, in an object literal written in the class, which the compiler reads when it writes the class's code:class Account { static readonly json = { rename: { userId: "user_id" }, optional: ["tags"] }; userId: i32 = 0; tags: string[] | null = null; }renamemaps a field to its key, a string literal of any bytes, so{"user_id":7}is decoded intouserIdanduserIdis encoded as"user_id"; the field's own name is then a key no field has. The key is dispatched by its length and bytes like any other, escapes and non-ASCII included (tests/link/std_json_typed/options.ts, "rename: escaped wire names decode").optionallists fields whose key may be missing: a missing one sets the field to its initializer, on a fresh object and on one decoded into again, so a reused object does not keep what an earlier input gave it ("optional: sets the missing fields back to their initializers"), and allocates nothing to do it. Every other missing key is stillJSON_DECODE_MISSING, and an optional key that is given is read like any other: once, and of its field's type. The member is not a field: it has no slot, the class's layout is what it would be without it, the program cannot read it, and a class without it is decoded exactly as before. It is the onestaticmember a class may declare, and its options are checked where the class is declared, whether or not any call decodes the class, each misuse with its own code:Misuse Code Case static jsonwithoutreadonlybut with a type,static readonly jsonwith a type annotation, or on a generic classNL2492 json_reject_options_mutable_typed,json_reject_options_annotated,json_reject_options_generic, and on a generic class nothing instantiates,json_reject_options_generic_unusedoptions, rename, a key or anoptionalentry not written as the literal it must be: a module constant, an array for the object, a string foroptional's arrayNL2493 json_reject_options_alias,json_reject_options_alias_whole,json_reject_options_not_object,json_reject_options_optional_not_arrayan option other than rename,optionalandpresenceNL2494 json_reject_options_unknowna name that is not one of the class's fields NL2495 json_reject_options_misspelt_rename,json_reject_options_misspelt_optionala rename to a value that is not a string NL2496 json_reject_options_rename_numbertwo fields with one key, a rename to another field's own name included, or a field renamed or made optional twice NL2497 json_reject_options_wire_collides,json_reject_options_wire_twice,json_reject_options_optional_twicean optional field with no initializer NL2498 json_reject_options_no_initializera rename to a key that is not valid UTF-8, a lone surrogate such as "\ud800"included, which no input could giveNL2499 json_reject_options_surrogatepresenceor one of its entries not written as an object literal of string literalsNL2493 json_reject_presence_not_object,json_reject_presence_not_literalpresencenaming a field, or a presence field, the class does not declareNL2495 json_reject_presence_misspelt_field,json_reject_presence_misspelt_flaga presence field that is not a boolean, isreadonly, is the field it records, records two fields, is recorded by a presence field of its own or is renamed or made optional; a field recorded twiceNL2592 json_reject_presence_not_boolean,json_reject_presence_readonly,json_reject_presence_itself,json_reject_presence_shared,json_reject_presence_of_presence,json_reject_presence_recorded,json_reject_presence_renamed,json_reject_presence_optional,reject_json_presence_twiceA spread in the literal is
`User.json` spreads another value into the typed JSON options: …, because the options are read from the class's text, where a spread names nothing (NL2711,reject_json_options_spread), and a computed key is refused by Phase 0 as it is anywhere (NL1041,json_reject_options_computed), a subclass byextends(json_reject_options_subclass), and a secondstatic readonly jsonis a duplicate member (reject_json_options_twice). The options are not a value:User.jsonnames a class where a value is due, and is refused as any class name there is (json_reject_options_read). Onlystatic readonly jsonmay leave out its type: writtenstatic json = { ... }, or as an instance field, it is an ordinary field to the parser and needs one (a type annotation is required,json_reject_options_mutable,json_reject_options_instance), and an instance field calledjsonwith a type is an ordinary field, read and written under its own name (tests/link/std_json_typed/options.ts, "instance json: an ordinary field decodes"). The options are not a member, so a field or a method calledjsonmay come before or after them, as in TypeScript, whose statics have a namespace of their own ("order: a json field before the options decodes", "order: a json method before the options decodes"). Under Node the literal is a static property of the class that nothing reads. -
presencerecords which keys were given. A third option names, for a field, abooleanfield of the same class that every decoding sets to whether the input gave that field's key; a key givennullis given:class UserPatch { static readonly json = { optional: ["email"], presence: { email: "hasEmail" } }; email: string | null = null; hasEmail: boolean = false; }So
{}leaveshasEmailfalse, and{"email":null}and{"email":"a"}set it true, whichoptionalalone cannot tell apart from a key given the initializer. Any field may be recorded, of any kind the decoder takes and optional or not (tests/link/json_presence). The presence field is off the wire: no key decodes into it,jsonEncodeleaves it out, it takes no part in the key-uniqueness rule, and no option renames it, makes it optional or records it. It is set when the object closes, from the per-field flag the decoder already keeps to report a duplicate or a missing key: onestore i1per recorded field, no allocation, and a class that records nothing compiles to the code it did before. Each object decoded sets its own, so a nested class, an array element and a map's value answer for their own keys, and a second decoding into one object answers for its own input. A decoding that fails may have set some presence fields and not others (none, when the input is not an object), so they are read only afterJSON_DECODE_OK. A JSON Merge Patch (RFC 7396) is applied with it: the patch is a class whose fields are all optional and recorded, andnullon a field that is not| nullisJSON_DECODE_TYPE, so the patch is refused before any of it is applied (examples/merge-patch.ts,tests/link/json_merge_patch). -
A
JsonNumberis a number's text.JsonNumber, fromnish/json-typed, holds a number as the bytes it was written in, so a field of it decodes and encodes any number byte for byte: an integer wider than au64, a fraction with more digits than anf64holds,1E+400and-0alike (tests/run.jsholds each line ofround-trip-options.txttoJSON.stringify(JSON.parse(line))with every number kept as its source). It is read as a number whole, not as an object, so anything but a number isJSON_DECODE_TYPE, and its length is bounded byJsonLimits.maxNumberByteslike any number's. Decoding into one reuses its bytes, soArena.used()stays flat over decodes into one object. It reads its text asnish/json-valuereads a number:text(),isI64()andi64(),isU64()andu64(), andf64(), the nearest double;setText(t)sets it whentis one JSON number and answers whether it was. A new one holds0. Its methods run under Node, andtests/run.jsholds them to the native lines (readers.ts). -
A
Map<string, V>is an object. Decoding empties the map the field holds once the object opens, then reads each key as a new string and its value as aV, made fresh for the key (a zero,"", the enum's first member,null, or a new object, array or map, which aMap's value class is made withnew C()for, so its constructor takes no parameters, NL2488,json_reject_map_value_constructor), andsets it, so the map's entries are the input's, in the input's order. A key the object gave already isJSON_DECODE_DUPLICATEbefore its value is read, whether it is spelled the same way or through an escape (docs/security/json.md, JSON-3). Encoding writes the entries in the map's own order, which is insertion order. That differs from a JavaScript object in one case: a key that reads as an array index ("1") keeps its place, whereJSON.parsewould move it first ("map: a key like an index keeps the input's order"). Unlike the rest of a decode, a map's entries are made anew each time, so decoding into a map allocates its keys and values. A value whose read fails is notset, so a map never holds a value the input did not give ("map: and the map holds only the entry read before it"). The maps of one decoding take at mostd.maxMapKeyskeys between them, 4,096 unless set, the keys of a map inside another map included, and the key past it isJSON_DECODE_MAP_LIMITbefore it is read ("map limit: one key past it: code", "map limit: 820 small maps, 4,100 keys, are refused");d.resetstarts the count again. At0or below no map may take a key, and{}is the one map value accepted. The globalMaphashes a string under a per-process random key, so a body cannot choose keys that collide, and this limit bounds what all the maps of one decoding hold together (docs/security/json.md, JSON-13). -
A tagged union is an object with its tag. A field, an array's element or the value of
jsonEncodemay be a discriminated union whose variants are all declared classes, and the union is written as the variant's object with its tag key first, the tag's value the name of its enum member, or for a string enum the member's string:enum Kind { Num, Neg } class Num { readonly kind: Kind.Num = Kind.Num; value: f64 = 0; } class Neg { static readonly json = { tag: "negate" }; readonly kind: Kind.Neg = Kind.Neg; arg: Expr; constructor() { this.arg = zero(); } } type Expr = Num | Neg; const zero = (): Expr => new Num(); const e = jsonDecodeUnion(d, zero()); // {"kind":"negate","arg":{"kind":"Num","value":2}} jsonEncode(w, e); // the same text backWith
enum Kind { Num = "num", Neg = "neg" }the same union writes"kind":"num", andNeg'stagstill writes"negate"(tests/link/json_enum_string). A variant'stagoption replaces its member's name or string on the wire, andrenamerenames the tag key as it renames any field. Every variant writes its tag under one key and as a different string, or the union is refused where a call reaches it (NL2721,json_reject_union_tag_key; NL2720,json_reject_union_same_tag); atagthat is not a string literal is NL2722 (json_reject_union_tag_literal), one on a class with no field whose type is one enum member is NL2724 (json_reject_union_tag_no_field), andtag: ""is the same as notag: the variant writes its member's name, or string. Atagon a class that has such a field but that no union holds is not refused and does nothing: the class compiles while no JSON call reaches it, and one that does refuses the field (below). A variant that is not a class, an interface, is NL2486 (json_reject_union_interface), the code of any typed JSON type that is not a declared class. A field typed with an enum member that no union tags a variant with is no tag, and is refused as any other field type is (json_reject_enum_member).jsonDecodedecodes into the object it is given and a union's variant is known only from the input, so a union is decoded withjsonDecodeUnion(d, prev), which answers the object it decoded into:previtself when the input's tag isprev's, reused as any nested object is, and otherwise a new variant made withnew V(). On a failure it answersprev, with the code ind.failed, at the top only: a union field, element orMapvalue already holds the new variant when the failure comes, decoded up to it, as any failed decode leaves what it read. Its type argument is inferred fromprevand never written;jsonDecodewith a union is`jsonDecode`'s value is `A | B`, a tagged union: …(NL2719,json_reject_union_decode), andjsonDecodeUnionwith a class is NL2723 (json_reject_union_not_union). A union field decodes into the object it holds the same way, and an array's element past its end is made fresh. The tag may come anywhere in the object: when it is the first key the variant is known at once, and otherwise the decoder reads ahead to it, stepping over the values before it, and rewinds, so each byte is read at mostmaxDepth+ 1 times (docs/security/json.md, JSON-17). A tag that is missing, not a string or no variant's isJSON_DECODE_TAG. A variant used alone, as a class, decodes and encodes its tag too, and a tag that is not its own isJSON_DECODE_TAG. Each union gets one decoding and one encoding function,.json.decode.union.<U>and.json.encode.union.<U>, which every place it appears calls, so a recursive union such asExprabove recurses through them (tests/link/json_tagged_union). AMap<string, Expr>field decodes each value through that same function and encodes it through the other. -
Decoding reuses what
outholds. A nested object is decoded into the object its field holds, an array's elements into the elements it has, with new ones made past its end and the rest dropped, and a string field keeps its string when the input's is equal. So decoding one shape into one object again and again allocates nothing:Arena.used()is flat over 1,000 decodes of a three-level object (tests/link/std_json_typed). A string that changes is a new string, since strings are immutable, and only that: it is made by oneString.fromBytes, straight from the input when the string has no escape and from its unescaped bytes when it has one, and those bytes are reclaimed at the call. A class the decoder has to make itself — an array's element past the end, or the object of a| nullfield that holdsnull— is made withnew C(), so its constructor takes no parameters:`jsonDecode` makes a `Pet` with `new Pet()`, and its constructor takes 1 parameter(s): an array's element and a `| null` field's object are made by the decoder with no arguments, so their class's constructor takes none(NL2488,json_reject_constructor,json_reject_constructor_nullable). Decoding assigns every field, so areadonlyone is refused:`Key.id` is `readonly`, and `jsonDecode` assigns every field of the class it decodes into(NL2487,json_reject_readonly). A decode stores what it makes into the object it fills, so a call inside ausing a = arena()block is the NL2424 of that block (json_reject_arena_block). An encoding only reads the object and keeps nothing, so it may run inside one (tests/link/std_json_typed). Two fields holding one object are decoded into it twice, the second winning. -
Encoding is
JSON.stringify's text for the same object, its keys in declaration order, which is also the order an object's keys come back in when it is decoded from text whose keys are in that order:tests/run.jsholds each line oftests/link/std_json_typed/round-trip.txt, decoded and encoded natively, toJSON.stringify(JSON.parse(line)). A full buffer is the writer'sJSON_WRITE_FULL, and an object that holds itself ends at the writer's depth limit withJSON_WRITE_TOO_DEEP, because the encoding stops at the first failure rather than recursing on. -
The TypeScript reading is class C (wp33-round-trip.md §2). Under Node the type arguments are erased, so there is no layout to write the code from, and the bodies in
std/json-typed.tsthat Node runs end the program withpanic("nish/json-typed: jsonDecode and jsonEncode are written by the compiler from the class's fields, and under Node the types are erased")rather than guess a field's type from the value it holds (tests/run.js, "under Node, jsonDecode panics"). The translation is the code the compiler writes, written as TypeScript for each class: a function per class of the same calls intoJsonDecoderandJsonWriter, which everything else in the module already runs under Node:tests/run.jsruns the decoder's readers there and natively and holds both to the same lines (tests/link/std_json_typed/readers.ts). A program that importsstd/json-typed.tsby path rather than asnish/json-typedis compiled as written and gets those bodies natively too.
Capabilities
Each builtin's label is the Capability: line of its doc comment in runtime/nish.d.ts; the rules are here.
Every builtin carries exactly one capability label, or a deliberate none, and the compiler works out which capabilities every function, module and package can reach — directly, through any chain of calls, across packages — with one witness call chain for each (wp35-capabilities.md). A program is refused for what it reaches only under a capability policy; without one, the set is a report.
| Capability | Builtins |
|---|---|
clock |
Date.now, monotonicNanos, wallClockNanos, statMtimeSync (tests/cases/caps_clock) |
entropy |
crypto.getRandomValues, Math.random (caps_entropy) |
env |
getenv, geteuid (caps_env) |
exit |
process.exit, and exit from nish:process (caps_exit) |
ffi |
any call to a declare function (caps_ffi) |
fs.read |
readFileSync, readFileSyncOrNull, readFileBytesSync, readdirSync, realpathSync, isDirectorySync, lstatOwnerModeSync, isExecutableSync, fileOpen, fileReadAt, fileSize, fileClose (caps_fs_read, caps_file_read) |
fs.write |
writeFileSync, appendFileSync, mkdirSync (caps_fs_write), writeFileBytesSync, renameSync, unlinkSync (caps_fs_replace), writeFileSyncMode, createFileExclusiveSync, mkdirSyncMode (caps_fs_mode) |
net |
every nish:net export (caps_net) |
process.spawn |
spawnSync, spawnSyncTo (caps_process_spawn), spawnPiped (caps_process_spawn_piped), spawnPipedWithin (caps_process_spawn_piped_within), spawn, kill, waitPid (caps_process_wait), setNice, setIdlePriority, setCpuAffinity (caps_process_nice) |
signal |
signalFd, readSignal (caps_signal) |
tty |
every nish:term export (caps_tty) |
unsafe |
the eight nish:unsafe exports: uncheckedGet, uncheckedSet, wrappingAdd, wrappingSub, wrappingMul, readCString, readStruct, cErrno (tests/link/caps_policy_deny_unsafe; "capabilities: each of the eight nish:unsafe exports carries unsafe at its call") |
Everything else is none: printing (console.*, write, writeError, writeBytes),
process.argv, process.platform and process.arch, availableParallelism and totalMemory (caps_host), panic, Arena.* and arena(),
Math.* other than random, the conversions, the constant-time builtins, secureZero, the
string, array, Map and Set surface, and nish/threads (caps_pure). A
nish: import carries the label of the builtin it renames.
-
A function is judged per instantiation:
pick<i32, double>andpick<string, firstLine>, orapply<lengthOf>andapply<(name) => ...>, are separate entries with separate sets (caps_generic,caps_fnarg). A module's set is the union over its functions, a package's over its modules, and the program's ismain's closure (tests/link/caps_package). A parallel body cannot reach a capability at all: every builtin that carries one writes memory the runtime shares (tests/link/caps_parallel,reject_caps_parallel_clock). -
A program is deterministic when it reaches none of
clock,entropy,env,ffi,fs.read,fs.write,net,process.spawnandtty: the same argv and stdin then give the same output.ttyis among them although stdin is an input a deterministic program may read, becausenish:termreads it against the clock (waitStdin's timeout,readStdin's "nothing yet") and asks the terminal itself.exitandsignalkeep a program deterministic (caps_exit,caps_signal). -
--emit-capabilities <file.json>writes the report: version, entry,deterministic, the program's set, then each package, module and exported function (and every instantiation of one) with its set and a witness chain per capability,{ "function", "at": "path:line:col", "calls" }per call, and then its"panics": every panic site the function has,{ "kind", "at": "path:line:col" }followed by the keys--emit-panicswrites after its position ("proven","callee","via","allowed"). Keys are in a fixed order, sets in the order of the table above, packages, modules and functions sorted, and every path relative to the entry's directory, so the file is byte-stable and diffable (.caps.jsonbeside eachcaps_*case).nish run --capabilities— or--capabilitieson a compile — prints one line on stderr,capabilities: fs.read (not deterministic)orcapabilities: none (deterministic), before the program runs, and leaves its stdout alone.--emit-capabilitieswithout a file is a usage error, exit 2, and so is either flag with--emit-ast, which stops before the checker; with--emit-checkedboth are answered beside the dump, and--fix, which answers before the program is analysed, takes neither.nish runrefuses--emit-capabilitiesas it refuses every product it keeps to itself.
The capability policy
--allow <cap>[,<cap>...] and --deny <cap>[,<cap>...], each repeatable,
and "nish": { "capabilities": { "allow": [...], "deny": [...] } } in the
root package.json (the one nearest above the entry, where noPanic is read)
are a capability policy, and a program whose entry reaches a capability
the policy does not grant is refused
(wp36-capability-policy.md). The same object's
dependencies key grants a dependency its gated capabilities, as a
dependency's capabilities says (NL3041, NL3042):
main.ts:6:36: error: `main` reaches `net`, which the capability policy does not grant (the root package's `deny`); the chain that reaches it: main.ts:6:36 calls dial, client.ts:5:10 calls netAddress
-
A denied capability is refused when it is reached. Giving
--allowor anallowlist makes the policy an allowlist: a reached capability it does not list is refused,exitandsignalincluded. The command line can only narrow the manifest: denials union and allowlists intersect. -
What is judged is every function outside code can call:
mainalone in a closed--linkbuild or undernish run, and in an open one (-oalone, a sidecar, adeclare function, wasm)mainand every exported function of every module, or every function at all under--no-strict-exports(tests/wordings/nl2459_policy_library,nl2459_policy_unexported). Each refused capability is one error (NL2459), reported in source order, spanned at the first call of the witness chain and naming every hop of it as--emit-capabilitieswrites them (tests/wordings/nl2459_policy_deny,nl2459_policy_allowlist;tests/link/caps_policy_manifest,caps_policy_two_denied). The command line can only narrow (caps_policy_cli_cannot_widen,caps_policy_allow_intersect,caps_policy_allow_narrowed,caps_policy_manifest_allow_cli_deny). -
unsafecan be denied —--deny unsaferefuses a program that reaches anish:unsafeexport anywhere in its closure (tests/link/caps_policy_deny_unsafe) — and never allowed: the import is the opt-in, so an allowlist does not refuse it and--allow unsafeis a usage error. -
unsafetravels like every other capability — to each caller, however far up, and to the module and package above it, including a dependency that importsnish:unsafewhen the entry package does not — and keeps a program deterministic ("capabilities:unsafereaches a caller two frames up, with the whole chain as its witness", "capabilities: a dependency that importsnish:unsafegivesunsafeto its package and to the entry that calls it"). No flag turns a check off for a whole program, sonish:unsafeis the only way to one and--deny unsafesees every unchecked site. -
A name that is no capability,
--allow unsafe, a capability both allowed and denied, and a directory scope (fs.read=<dir>) without--sandbox, which alone enforces one (Directory scopes under The sandbox), are usage errors, exit 2. Inpackage.jsonthey are errors where the manifest writes them: the wrong shape ofcapabilities,allowordenywritten twice included (NL3032,tests/link/caps_policy_manifest_shape,caps_policy_manifest_repeated), a name that is no capability (NL3033,caps_policy_manifest_unknown),unsafeunderallow(NL3034,caps_policy_manifest_unsafe) and a capability under both (NL3035,caps_policy_manifest_both). -
The
"nish"field is read strictly, as the no-panic scope says: a key is decoded asJSON.parsedecodes it, so a policy written with an escape is honoured (caps_policy_manifest_escaped_policy,caps_policy_manifest_escaped_entry), and anything the reader could otherwise not vouch for — a second"nish"orcapabilities, a key other thannoPanic,capabilities,host,compatandnetRings— is NL3036 (caps_policy_manifest_repeated_nish,caps_policy_manifest_repeated_policy,caps_policy_manifest_nish_unknown_key). -
A policy changes no byte of the IR: a program inside it compiles and runs as
it does with none (
tests/link/caps_policy_granted,caps_policy_manifest_granted).nish runtakes both flags and refuses before it links;--fixand--emit-astrefuse them.
A dependency's capabilities
A dependency — a package loaded from node_modules — may reach ffi,
fs.read, fs.write, net, process.spawn and unsafe, the gated
capabilities, only when the root
package.json grants them to it by name; it reaches every other capability
without one (wp40-dependency-capabilities.md):
{ "nish": { "capabilities": { "dependencies": { "pkg_http": ["net"] } } } }
A dependency that reaches one it is not granted is refused (NL2576), in the dependency, naming the package, the capability and the function:
node_modules/pkg_http/src/index.ts:6:10: error: the dependency `pkg_http` reaches `net` through `dial`, which the root package does not grant it; grant it with "nish": { "capabilities": { "dependencies": { "pkg_http": ["net"] } } } — the chain that reaches it: node_modules/pkg_http/src/index.ts:6:10 calls netAddress
-
A dependency's capabilities are the union over every function its
modules define, exported or not, called or not
(
tests/link/caps_dep_unexported), and what it reaches through another dependency is its own: granting the package that dials refuses the one that calls it (caps_dep_transitive). One error per package and capability, spanned at the first call of the witness chain from the package's first function, in source order, that reaches it (caps_dep_order; one refused and one granted program per capability,caps_dep_net_refused…caps_dep_unsafe_granted). -
The standard library is never refused, and what decides is where a module
was resolved, not its package's name: a dependency that calls itself
nishis judged like any other (caps_dep_std_not_refused,caps_dep_named_nish). The ungated capabilities need no grant (caps_dep_ungated). -
A function argument counts where it is instantiated: the root's function
passed to a dependency's template makes the instantiation, which is the
dependency's, reach what that function reaches, so the dependency is
refused for it (
caps_dep_fnarg_callback). -
Only the root manifest grants: no flag does (
--allow netleaves the dependency refused,caps_dep_cli_cannot_grant), nor the dependency's own manifest (caps_dep_self_grant), and a grant never widens the root's own policy — a capability the root denies is refused by NL2459 whoever is granted it (caps_dep_root_deny_wins). -
dependenciesis read as strictly as the rest of the field: a value that is not an object, a package named twice, a grant that is not an array and an entry that is not a string are NL3041 (caps_dep_manifest_not_object,_repeated,_not_array,_non_string), a name that is no capability is NL3033 (caps_dep_manifest_unknown) and one that is not gated NL3042 (caps_dep_manifest_ungated).unsafemay be granted, keys are decoded as JSON, and a grant to a package the program does not load is no error (caps_dep_manifest_unsafe,_escaped_name,_unloaded). -
fs.readandfs.writeare gated: a dependency that reads or writes a file needs its grant like one that dials (caps_dep_fs_read_refused,caps_dep_fs_write_refused).
A grant entry may carry an argument, capability=argument, which narrows
it: a directory for fs.read and fs.write, a program for process.spawn,
a numeric host, or a Unix socket's absolute path, for net. An entry
without one grants the capability whole, as before:
{ "nish": { "capabilities": { "dependencies": {
"pkg_tool": ["fs.read=./data", "process.spawn=git", "net=127.0.0.1"]
} } } }
-
A directory is resolved from the root package's directory and must exist when the program is built (NL3047); a program is found on
PATH, or as a path from that directory (NL3046); a host is an IPv4 or IPv6 address, the formnish:nettakes (NL3045).ffi,unsafeand the ungated capabilities take no argument (NL3048; all four incaps_dep_manifest_arg_refused). A capability may be named more than once, one argument each. -
What the compiler can see it checks: a string literal a dependency hands
readFileSync,writeFileSync,spawnSyncornetAddressoutside its grant is refused (NL2594), at the literal:node_modules/pkg_conf/src/index.ts:2:38: error: the dependency `pkg_conf` hands `readFileSync` "/etc/passwd", which its `fs.read` grant does not cover: it is granted fs.read=data; name it in the grant, or grant `fs.read` whole(
caps_dep_arg_fs_too_narrow,_spawn_too_narrow,_net_too_narrow; the grant that covers each,caps_dep_arg_granted). -
What it cannot see — a computed path, program or host — the build leaves to
--sandbox, which scopes Landlock to the granted directories and programs (The sandbox,tests/link/sandbox_dep_*). Hosts have no run-time check: seccomp cannot read the address a socket is given, so anet=<host>grant binds only the literals the build sees. -
The lockfile records the arguments (
"pkg_conf": ["fs.read=data"]). A grant past them — a new directory, program or host for a capability the row records narrowed — is NL2593 until the lock is updated; a row that records the capability whole accepts any argument (caps_lock_args). -
@nish_sandboxtakes a fifth operand for the programs aprocess.spawn=<program>grant allows, null when none is named, and a sixth, the number of restricted rings to make before the filter (WP63 S3; 0 unlessmainreachesnetwithNET_LISTEN_RINGand notprocess.spawn):call void @nish_sandbox(i32 32, i32 1, i8* null, i8* null, i8* null, i32 0). -
process.spawn=exec:<program>grants the program asprocess.spawn=<program>does, and lets it start others: under--sandboxa granted program is held to its own start (no program it starts may exec again), and anexec:one is not, so its subtree may run anything the allowlist's EXECUTE reaches, its loaders included. The prefix is read before the program is resolved:process.spawn=exec:with no program is NL3046 (caps_dep_manifest_spawn_exec_refused); the literal check treatsexec:<program>as<program>(caps_dep_arg_spawn_exec_too_narrow,caps_dep_spawn_exec_granted); a program granted both ways isexec:; and the lockfile records the prefix, so adding it is NL2593 (caps_dep_lock_spawn_exec_widened). The program is listed with three leading slashes, the same file to the kernel. Fortests/link/sandbox_dep_exec_optin, grantedexec:envandtrue, on an x86_64 glibc machine (illustrative: the paths, the loader and the length are the machine's;tests/run.jspins only the///entry forenv):@.sandbox.exec = private unnamed_addr constant [78 x i8] c"///usr/bin/env\00/usr/bin/true\00//usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2\00\00"
The capability lockfile
--update-capability-lock writes nish-capabilities.lock.json beside the
nearest package.json above the entry (beside the entry when there is none),
recording every capability each dependency reaches, gated or not. While the
file exists, every build is held to it:
{
"version": 1,
"packages": {
"pkg_http": ["net"]
}
}
A dependency that reaches a capability its row does not record is refused (NL2577), in the dependency, with the same witness chain as NL2576:
node_modules/pkg_http/src/index.ts:6:10: error: the dependency `pkg_http` reaches `net` through `dial`, which `nish-capabilities.lock.json` does not record for it; review the chain, then record it with `--update-capability-lock` — the chain that reaches it: node_modules/pkg_http/src/index.ts:6:10 calls netAddress
-
Only growth is an error. A row that records more than the dependency
reaches, or a row for a package the program no longer loads, is accepted
(
tests/link/caps_lock_shrink,caps_lock_stale); a dependency with no row is held to an empty one (caps_lock_no_row,caps_lock_no_row_pure). One error per package and capability, in NL2576's order (caps_lock_growth). -
The comparison runs after both policies: a build the root's policy or the
dependency grants refuse is refused by them, and
--update-capability-lockon it writes nothing (caps_lock_refused). The flag skips the comparison and rewrites the file, even a malformed one. -
The file is byte-stable: packages sorted, capabilities in the fixed order of
the report, two-space indentation, one trailing newline,
[]for a package that reaches nothing. Neither the standard library nor the root package has a row. -
It is read strictly: text that is not JSON, a
versionother than1, a key other thanversionandpackages, a name that is no capability, and a key or capability written twice are NL3043, spanned in the lockfile (caps_lock_bad_json,_version,_key,_capability,_repeated). -
The lockfile changes no byte of the IR.
nish runtakes the flag, and checks the program against the file before it looks in its cache, so the file is not part of the cache key and a cache hit is held to it too.--fixand--emit-astrefuse the flag.
--deterministic
--deterministic, on a compile or nish run, promises that the program's
f64 and f32 results are the same bits natively on x86-64 and AArch64, on
wasm32 and in the --emit-ts reading under Node
(wp50-wasm-host.md §9):
-
Math.sin,cos,exp,logandpowlower tostd/math.ts'ssinF64,cosF64,expF64,logF64andpowF64on every target, as on a page module, and a float%tofmodF64(anf32widened and the exact result narrowed back). The--emit-tsreading imports those functions, andtanF64,atan2F64andhypotF64, from./nish/math.tsin place ofMath.+ - * /,sqrtand the exact builtins need nothing: they are correctly rounded everywhere, and every C unit and the program's IR are compiled with-ffp-contract=offwith or without the flag. -
f64ToBitsandf32ToBitsgive the canonical quiet NaN,0x7ff8000000000000and0x7fc00000, for every NaN (quietBitsF64andquietBitsF32); targets differ in the sign and payload of a NaN an operation makes. -
A program that reaches a nondeterministic capability —
clock,entropy,env,ffi,fs.read,fs.write,net,process.spawnortty— is refused with the chain that reaches it, as the capability policy refuses one: NL3052 the file system, NL3053 another machine or program (net,process.spawn), NL3054 C code or a host import (ffi), NL2700 the clock and the random source, NL2701 the environment and the terminal. -
Number(s),parseFloat(s)andparseInt(s)read a string with one grammar on every target, the one--emit-tsreads: a0xinteger, read bystd/math.ts'snumberExactandparseFloatExact(which the flag calls in their place) and correctly rounded on any libc, and no hex float, soNumber("0x1p3")is NaN andparseFloat("0x1p3")is 1 where a native build without the flag letsstrtodread 8 (deterministic_parse). Every decimal spelling already agreed: glibc, musl and Node round them all correctly.
main.ts:2:15: error: `main` reaches `clock`, which `--deterministic` refuses: the clock and the random source answer differently on every run; the chain that reaches it: main.ts:2:15 calls Date.now
The float corpus, tests/differential/corpus/float/, holds every operation
and Math builtin over ±0, subnormals, the normal extremes, ±Infinity, NaN,
integers near 2^53, multiples of π/4, NaN payloads read through
f64ToBits and f32ToBits, and 94 strings through Number, parseFloat and
parseInt to one expected.txt on each target
(tests/run.js, sections deterministic and float-corpus; the
reject_deterministic_* cases).
The sandbox
--sandbox holds a running native program to the capabilities its main
reaches. The policy above refuses at compile time a program that reaches more
than it grants; the sandbox makes the kernel refuse, at run time, what the
program's code does not reach, ffi callees and constructors included
(wp37-os-sandbox.md,
security/sandbox.md):
@.sandbox.preinit = internal constant void ()* @.sandbox.apply, section ".preinit_array", align 8
@llvm.used = appending global [1 x i8*] [i8* bitcast (void ()** @.sandbox.preinit to i8*)], section "llvm.metadata"
define internal void @.sandbox.apply() {
entry:
call void @nish_sandbox(i32 0, i32 1, i8* null, i8* null, i8* null, i32 0)
ret void
}
define noundef i32 @main(i32 noundef %argc, i8** noundef %argv) #1 {
entry:
call void @nish_sandbox_check()
-
The flag.
--sandboxis strict;--sandbox=best-effortruns on a kernel that lacks part of the confinement and says in one line on stderr what it could not withhold. Any other mode is a usage error, exit 2. It is accepted by--link, bynish run(before the file) and on the IR-only route, alone or beside the flags that write a file next to the IR; the.llthen carries the call, and linking it by hand keeps the confinement (tests/cases/sandbox_hello).--emit-ast,--emit-checked,--emit-ts,--fix,--emit-napiand--emit-napi-asyncrefuse it, exit 2, since none writes a program with amainto confine, and so does an entry module that declares nomainand holds no top-level code to run as one (--sandboxon a script confines themainits code becomes).--sandboxand--sandbox=best-efforton one command line are refused, exit 2; the same one twice is one. -
The call. The first operand is
main's capability mask, numbered as--emit-capabilitiesnumbers the bits (clock0 …tty10,unsafe11), the second1for strict and2for best-effort; the twoi8*operands are thefs.readandfs.writedirectory scopes below,nullwhen there are none.nish_sandboxruns from.preinit_array, after the loader and before any constructor of the executable, anffiobject or a shared library, and applies, in order:PR_SET_NO_NEW_PRIVS, a drop of every Linux capability (each set read back), a Landlock ruleset masked to the ABI the kernel answers, and a seccomp filter.maincallsnish_sandbox_checkfirst, which refuses to run a program whose C library never ran.preinit_array(musl), with exit 126 and one line:nish: --sandbox: the C library did not run .preinit_array, so the program was not confined. -
What each capability's absence withholds. Without
fs.read, Landlock withholds reading files and directories; withoutfs.write, writing, making, removing, renaming and linking (within one directory too) and truncating, and seccomp answersEPERMtochmod,chown,utimensat,setxattrand their siblings, which Landlock does not decide. Withoutprocess.spawn,execve,execveat,fork,vforkand aclonewithoutCLONE_THREADkill the process,clone3answersENOSYS, and Landlock withholds executing files and deviceioctls (ABI 5); with it, executing and reading files and deviceioctls are granted beneath/, since a child must load its binary and libraries, so a program that reachesprocess.spawncan read any file the user can whether or not it reachesfs.read. Withoutnet,socket,socketpair,connect,bind,listen,acceptandaccept4kill it. Withnetbut withoutfs.write, a pathnameunixListenorunixListenModeis-13: Landlock counts making the socket file as a write (MAKE_SOCK), while an abstract name binds. The sandbox uses no Landlock right that covers connecting to a pathname socket, sonetalone still letsunixConnectreach any pathname listener whose file and directories the user may reach; only abstract names outside the sandbox are scoped, from ABI 6.ffi,unsafe,tty,clock,entropy,env,exitandsignalwithhold nothing of their own; the ABI 6 scopes under Always apply whatevermainreaches. -
Directory scopes. Under
--sandbox,--allow fs.read=<dir>and--allow fs.write=<dir>grant the capability beneath<dir>alone. The directory is resolved, symlinks and all, when the program is built, and must exist then, or it is a usage error, exit 2:compile: `fs.read=<dir>` names no directory; a scope is a directory that exists when the program is built. Any other capability scoped, or a scope in--deny, is refused, exit 2, and so is a scope without--sandbox, as before. A scope narrows the policy like any--allowlist, so the same list names every other capability the program reaches. A capability also granted whole is granted whole. The scopes are emitted as one constant per capability, each directory ended by a NUL and the list by a second:@.sandbox.read = private unnamed_addr constant [N x i8] c"/srv/data\00\00", passed asnish_sandbox's third (fs.read) and fourth (fs.write) operand,nullwhenmaindoes not reach the capability. Landlock grants the scoped access beneath each directory and nothing else, except that amainreachingprocess.spawnkeeps reading every file beneath/(READ_FILE, so thatexeccan load a program), scope or not;chmod,chown,utimensat,setxattrand their siblings still answerEPERMinside anfs.writescope, since seccomp cannot read a path. A main that reachesffiis widened by the policy'snet,fs.read,fs.writeandprocess.spawngrants, a scope included, since foreign code reaches them without the checker seeing it. A directory cannot hold a comma. -
The CPU count. Before it confines anything,
nish_sandboxreads the CPU count (the affinity mask and the cgroup quota) and keeps it as the thread pool's ceiling, since withoutfs.readthe quota under/sys/fs/cgroupcannot be read afterwards.NISH_NUM_THREADSstill lowers the pool.availableParallelism()reads the count again on each call and is not bounded by that ceiling, so in such a program it no longer sees the quota. -
Always.
ptrace,process_vm_readv,process_vm_writevandpidfd_getfdkill; io_uring kills withoutnet, and withnetbut notprocess.spawnonlyio_uring_enteris let through, on the premade rings' descriptors, matched by descriptor number (see the premade rings below); on x86-64 a system call numbered with bit 30 (x32) kills, and a call from any other architecture's entry kills. A killed process ends bySIGSYS, exit status 159 from a shell. From Landlock ABI 6 a sandboxed process cannot signal or reach the abstract sockets of a process outside it. -
The premade rings. With
netgranted andprocess.spawnnot, a program whosemainreachestcpListenFlagswithNET_LISTEN_RINGgets its rings before the filter loads:nish.netRingsin the rootpackage.json(an integer from 1 to 64, default 1; anything else is NL3058) restricted rings, each taking onlyACCEPT,RECV,SEND,CLOSE,ASYNC_CANCELandNOP, with the submission flagsFIXED_FILE,BUFFER_SELECTandCQE_SKIP_SUCCESS; any other submission completes with-13. Ring listeners take them as they fill: once every ring made is in use, a listener whose rings are full answers-23fromtcpAcceptuntil a connection closes, a ring listener opened after that serves descriptors, and a kernel or limit that refuses them leaves the process on descriptors for good, whichnetRingState()reports. One more ring, the keeper, takes onlyNOPand is entered by nothing; a ring a closed listener held has the keeper put at its descriptor number, so no premade ring's number is ever free.close,dup2anddup3naming a ring's or the keeper's number as the descriptor they close or replace kill, except adup3from the keeper, which is how the runtime puts the keeper at a freed ring's number, andclose_rangefails withENOSYS, as on a kernel older than 5.9; on every other descriptor they work as before (NURING-36 indocs/security/net-uring.md). -
Refusals. A program whose sandbox cannot be applied prints one line,
nish: --sandbox: ..., and exits 126 beforemainruns. Strict refuses a kernel without Landlock, and one below ABI 3 whenfs.writeis not granted, sinceTRUNCATEcannot be withheld there, or when it is granted only beneath a directory, since below ABI 3 that grants nothing beneath/; best-effort runs on both, naming what it could not withhold (without Landlock, files are not confined at all). A scope directory missing at run time is refused by strict,nish: --sandbox: cannot open the directory <dir> (ENOENT); best-effort leaves its rule out, so nothing beneath it is granted, and names it once. Neither mode refuses a kernel withoutIOCTL_DEV(ABI 5) or the signal and socket scopes (ABI 6); best-effort names them. -
Targets. A darwin target, given or the host's, is refused (NL3044,
tests/wordings/nl3044_sandbox_darwin). The wasm and WASI targets (wasm32,wasm32-wasi) accept the flag and emit no call, since the host already holds a module to its imports;--profile wasmrefuses IR built with--sandboxfor the host. A.llbuilt with--sandboxand linked on a platform other than Linux refuses to run, exit 126. -
No byte of a program's IR moves without the flag: the entry, its
@llvm.usedslot and the two declarations are emitted only under--sandbox("sandbox: without the flag the IR is the sandboxed IR less the sandbox's lines").--runtime-decls, which lists every declaration the runtime answers, lists the two whatever the flag. -
Reserved names. The runtime calls
prctlandsyscallfrom C, so in every build, sandboxed or not, adeclare functionof either name is refused as a reserved C symbol (NL3037).

