The standard library

Nish ships a standard library of modules written in Nish. You import one by its nish/ name:

import { Suite } from "nish/testing";

nish/<module> always means the library that came with the compiler you are running, whatever directory your file is in. A library module is source, not a prebuilt library: it is compiled with your program, under the same rules, and what you import but never call is dropped from the binary. A name the library does not have is an error that lists the modules it does.

The language itself has few string helpers — there is no split, trim or RegExp — so nish/text is where you look first, and nish/regex when a pattern is what you need. Map and Set need no import at all (Maps and sets), and the built-in modules that wrap the operating system (nish:fs, nish:process, nish:io, nish:net) are part of the language rather than this library; see Builtin modules.

nish:process is where a program starts a child without waiting for it (spawn), signals it (kill) and reaps it (waitPid), and where it asks how many cores and bytes of memory the machine has (availableParallelism, totalMemory) to size a pool of workers; see Children and the machine. A batch job steps aside for interactive work with setNice(n) (the nice value, 0..19, lower-only: -22 outside that range and -13 below the current value, even as root) or, on Linux, setIdlePriority() (SCHED_IDLE: the CPU only when nothing else wants it).

setNice and setIdlePriority each answer 0 or a negative errno and never throw; setIdlePriority is -95 on Darwin. An unprivileged process can never leave SCHED_IDLE, and an idle one can starve on a busy machine, so keep a normal-priority fallback for work that must finish, and call either at the top of main, before any thread starts. See Scheduling.

spawnPiped(argv, stdin, stdout, stderr) runs a child with every stream in memory: the child reads stdin's bytes and then end of file, and what it writes is appended to stdout and stderr. It answers the exit status (128 plus the signal for one killed by a signal), -1 when the child cannot start, and -2 when either output passes 16 MiB, after which the child is killed. It is how a program hands a child a secret, since a child's argument vector and environment are readable by other processes and a file outlives the run:

const token = "ghp_example"
const request = `protocol=https\nhost=github.com\nusername=bot\npassword=${token}\n\n`
const input: u8[] = []
for (let i = 0; i < request.length; i++) {
  input.push(toU8(request.charCodeAt(i)))
}
const out: u8[] = []
const err: u8[] = []
const status = spawnPiped(["git", "credential", "approve"], input, out, err)
if (status !== 0) {
  console.log(`git credential failed (${status}): ${String.fromBytes(err, 0, err.length)}`)
}

spawnPipedWithin(argv, stdin, stdout, stderr, timeoutMs) is the same run with a deadline, for a child that may never answer. When timeoutMs milliseconds pass before the child exits it answers -3, kills the child, and keeps what the child printed so far in stdout and stderr. A negative timeoutMs means no deadline. Only the child itself is killed: a process it started that still holds its output keeps the run going until the deadline, so a child that leaves one behind answers -3 even when it exited 0.

const out: u8[] = []
const err: u8[] = []
const none: u8[] = []
const status = spawnPipedWithin(["git", "ls-remote", "origin"], none, out, err, 10000)
if (status === -3) {
  console.log("git ls-remote took longer than 10 seconds")
}

Both are process.spawn in the capability report, and refused when compiling for WebAssembly. See Directories and subprocesses.

spawnWithFds(argv, fds) starts a child as spawn does and hands it the descriptors in fds: fds[0] is the child's descriptor 3, fds[1] its 4, and so on, which is where a program started by a supervisor looks for a listener it did not open. It answers the pid, or a negative errno: -22 for more than 64 descriptors or a negative one, -9 for one that is not open. No other descriptor of the program reaches the child but its three streams, and the child shares each one it is handed, offset and flags included. Reap it with waitPid:

import { spawnWithFds, waitPid } from "nish:process"
import { tcpListen } from "nish:net"

const listener = tcpListen("127.0.0.1", 8080, 64)
const pid = spawnWithFds(["./worker"], [listener])
if (pid > 0) {
  waitPid(pid, 0)
}

It is process.spawn in the capability report too. See Children and the machine.

nish:fs is where a program replaces a file in one step: write the new contents beside the target, then renameSync the temporary over it, so a reader sees the old file or the new one and never half of either. renameSync, unlinkSync, writeFileBytesSync and the mode-taking writes below answer a number instead of exiting: 0 (or the bytes written) on success and a negative errno on failure, such as -2 for a missing path or -18 when the two names are on different filesystems, so the program decides what to say.

Bytes that are not text, a u8[] from readFileBytesSync or a codec, go out as they are through writeFileBytesSync and, on stdout, writeBytes from nish:io; see File I/O. A native string built by String.fromBytes carries any bytes, and write and writeFileSync send it as it is, but under --emit-ts String.fromBytes decodes UTF-8 and an invalid sequence becomes U+FFFD, so writeBytes and writeFileBytesSync are the route that is exact everywhere.

A file too big to hold whole is read a window at a time. fileOpen(path) answers an i32 descriptor for a regular file, or a negative errno (-2 missing, -21 a directory); fileReadAt(fd, buf, off, len, pos) reads at most len bytes from byte pos (an i64) into buf at off and answers the count, 0 at the end; fileClose(fd) gives the descriptor back, once. The buffer is yours, so memory stays the size of the window:

export const main = (): number => {
  const fd = fileOpen("big.log");
  if (fd < 0) {
    console.log(`open failed: ${fd}`);
    return 1;
  }
  const buf: u8[] = new Array<u8>(65536);
  let pos: i64 = toI64(0);
  while (true) {
    const n = fileReadAt(fd, buf, 0, buf.length, pos);
    if (n < 0) {
      console.log(`read failed: ${n}`);
      break;
    }
    if (n === 0) {
      break;
    }
    pos = pos + toI64(n);
  }
  fileClose(fd);
  console.log(`read ${pos} bytes`);
  return 0;
};

A read may come back short, so loop on the count rather than on fileSize(fd). Serving a file sends one over HTTP this way. All four need fs.read, work natively, under Node and under --profile wasi, and are refused on freestanding wasm32 (NL2718); see File I/O.

Writing a secret to disk, a key or a token, takes three things writeFileSync does not give. writeFileSyncMode(path, data, 0o600) leaves the file at exactly the mode you name, set before the first byte is written, so the umask cannot widen it and a file that was 0644 is narrowed before it holds the secret.

A symbolic link at path is refused with a negative errno (-40 on Linux) and the file it names is left alone, so nobody can point your write somewhere else. And to replace a secret in one step, write it at 0600 to a temporary beside the target and renameSync the temporary over it: a reader then sees the old secret or the new one, never half of either.

createFileExclusiveSync(path, data, mode) is the same write as writeFileSyncMode, but it fails with -17 when the name already exists, so of any number of processes racing for one path exactly one wins: a lock file, released with unlinkSync. mkdirSyncMode(path, 0o700) makes a directory only you can enter; the umask still applies to a directory, which only ever narrows it. A mode past 0o777, the setuid, setgid and sticky bits included, is -22 and nothing is made. The numbers are the platform's errno values (Linux's shown).

Module What it gives you
nish/testing Suite, a test runner whose report is the exit code
nish/text split, splitLines, splitWhitespace, trim, trimStart, trimEnd, contains, replaceAll, asciiUpper, asciiLower, padStart, padEnd, firstDifference, codePointLength, indexOfAny, splitLimit, toFixed
nish/regex regexCompile: JavaScript-syntax regular expressions, matched in time linear in the input
nish/json jsonField and jsonFields to read the fields of one flat JSON object; JsonWriter to write JSON into a byte buffer, minified or indented, with an optional strict mode for NaN and the infinities
nish/json-tokenizer JsonTokenizer, which checks any JSON text and reads it one token at a time, with limits
nish/json-typed jsonDecode and jsonEncode: read a JSON object into a class and write one back, with renamed and optional keys, which keys were given, tagged unions, and lossless numbers
nish/json-value JsonDocument: any JSON document as a tree to read, change and write back, with every number kept exactly as written
nish/yaml parseYaml: a YAML 1.2 stream as documents, nodes and diagnostics, each with its byte range; pathNode to find a key, through aliases and merge keys; nish/yaml-lines for line:column; stringifyYaml in nish/yaml-write to write a JSON value as YAML; YamlJsonLoader in nish/yaml-json to load a document into a JsonDocument with its ranges
nish/pair Pair<A, B>, for a function that returns two values
nish/map getOrInsert and reserve for Map
nish/base64 base64Encode and base64EncodeInto: standard base64 with = padding, into a string or straight into a byte buffer
nish/bytes getU32LE, setU16BE and the rest: integers and doubles read from and written into a u8[] at an offset, in either byte order; skipBlank, skipDigits, skipJsonText, skipJsonPlain and copyBytes for the runs a parser skips and a writer copies
nish/temporal PlainDate, PlainTime, PlainDateTime, Instant and Duration: JavaScript's Temporal, in the ISO calendar, natively and under Node
nish/random Rng, a seeded generator of random numbers and dice (nextInt) that answers the same under Node
nish/cli HelpMenu and styleFor: a --help menu in the terminal theme's own colours, plain when piped; nish/cli-flags' FlagTable parses the flags, positionals and subcommand that menu lists
nish/cli-args Cli and parse: a command tree, its flags and arguments, help and did-you-mean errors
nish/release releaseLatest and releaseCheckCached: whether a newer release of a project on GitHub is out, asked at most once a day
nish/term Screen, Canvas and KeyDecoder: a terminal UI drawn as cells or half-block pixels, sent as only what changed, and what it reads — keys, mouse, paste, focus and the terminal's replies — decoded as TermEvents; KeyState for which keys are held
nish/term-io runTerminal and openTerminal: open the terminal, find out what it can draw, and run a frame loop whose rate and detail adapt to the link; nish/term-probe, nish/term-rate and nish/term-pixels are its pure parts
nish/input StepInput: what one fixed step reads from its player, the events since the last step and the keys held down, with no terminal, I/O or clock; isDown reads a key
nish/raster Drawing into a frame of 0xRRGGBB pixels: colours, gradients, rectangles, anti-aliased discs, glows, 5×7 pixel text, trails and particles, for a terminal's framebuffer and a page's alike
nish/synth Short sounds made from nothing: square, triangle and noise voices with a pitch slide and a decay, as f64 samples
nish/host/game The frame, input, sound queue and samples a wasm32 game shares with its page, as module storage
nish/sqlite openDatabase, Database and Statement: a SQLite database, with statements prepared once and rows read without allocating
nish/threads parallelMapInto, parallelMap, parallelFrom, parallelReduce, scope, Mutex, Channel
nish:secret Secret<T>: key material the compiler keeps from leaking, and wipes
nish/crypto/* SHA-2, HMAC, HKDF, ChaCha20-Poly1305, AES-GCM, X25519, Ed25519, P-256 ECDSA, X.509, trust anchors, base64url, bcrypt
nish/net/* TLS 1.3, QUIC, HTTP/1.1, HTTP/2, HTTP/3, WebSocket (both sides of its handshake), an HTTP router, a PostgreSQL client
nish/compress/* gzip, zlib and raw DEFLATE, both ways, into buffers you own or streamed; CRC-32 and Adler-32; nish/net/http-compress for Accept-Encoding and gzip responses
nish/net/url urlQueryGet and urlDecode for one value; UrlPathSegments, UrlQuery and percentDecode to read a request target in place, allocating nothing; urlParse, urlResolve and urlOrigin for a whole URL
nish/net/dns-message dnsBuildQuery and dnsParseReply: one DNS question as bytes, and the A or AAAA addresses a reply gives it, following its CNAME chain, with every count and pointer bounded
nish/net/dns-config dnsParseResolvConf and dnsParseHosts: /etc/resolv.conf and /etc/hosts read into fixed tables; dnsHostsFind and dnsParseAddress
nish/net/dns-resolver DnsResolver: a DNS stub resolver with no I/O of its own, which says what to send and reads what came back; it builds for wasm32
nish/net/dns DnsClient, the resolver on your own pollWait loop, and resolveHost, a blocking lookup for a CLI or start-up

nish/testing

A Suite collects named checks and turns them into a report and an exit code. A function cannot be a value in Nish, so there is no test("name", () => ...): you call one assertion per check, and each answers whether it passed.

import { Suite } from "nish/testing";

const sumOf = (xs: i32[]): i32 => {
  let total = 0;
  for (const x of xs) {
    total = total + x;
  }
  return total;
};

export const main = (): i32 => {
  const t = new Suite("sums");
  t.eqI32("four numbers", sumOf([3, 9, 4, 9]), 25);
  t.eqI32("no numbers", sumOf([]), 0);
  return t.done();
};
$ nish run sums.ts
PASS  four numbers
PASS  no numbers
sums: 2 passed, 0 failed, 0 skipped

done() prints the summary and answers 0 when nothing failed and 1 otherwise, so returning it from main makes the program's exit status the verdict. There is one assertion per type — eqI32, eqI64, eqBool, eqStr and the rest — plus ok(name, condition), contains, containsAll, eqLines for two arrays of lines, and skip(name, reason). Because an assertion returns a boolean, a test can stop before a read that would panic: if (!t.eqI32("length", xs.length, 3)) { return t.done(); }.

nish/text

The string operations a program would otherwise write by hand.

import { contains, replaceAll, splitLines, splitWhitespace, trim } from "nish/text";

export const main = (): i32 => {
  for (const line of splitLines("first line\n  second  line \n")) {
    console.log(`[${trim(line)}] has ${splitWhitespace(line).length} words`);
  }
  console.log(replaceAll("a-b-c", "-", "+"));
  console.log(`${contains("haystack", "st")}`);
  return 0;
};
$ nish run text.ts
[first line] has 2 words
[second  line] has 2 words
a+b+c
true

split(text, sep) is JavaScript's text.split(sep): a separator at either end or two in a row give empty strings, "" splits to [""], and a separator that never occurs gives [text]. An empty separator panics (split: an empty separator) rather than splitting into units, which are bytes in a compiled program and UTF-16 units under Node. The answers are the same both ways for any text and any non-empty separator.

splitLimit(text, sep, limit) is JavaScript's text.split(sep, limit): it keeps the first limit pieces and drops the rest, so splitLimit("a,b,c", ",", 2) is ["a", "b"] and a limit of 0 gives []. A negative limit keeps every piece, as JavaScript's does, so splitLimit(text, sep, -1) is split(text, sep). An empty separator panics here too (splitLimit: an empty separator).

toFixed(x, digits) is JavaScript's x.toFixed(digits), which Nish cannot write as a method: x with digits digits after the point, rounded on the exact value the f64 holds. That is why toFixed(1.005, 2) is "1.00": 1.005 is stored a little below itself. A tie rounds away from zero, so toFixed(2.5, 0) is "3" and toFixed(-2.5, 0) is "-3".

import { toFixed } from "nish/text";

export const main = (): i32 => {
  const price: f64 = 19.999;
  console.log(`total: ${toFixed(price * 3, 2)}`);
  console.log(toFixed(-0.0001, 2));
  return 0;
};
$ nish run fixed.ts
total: 60.00
-0.00

digits is 0 to 100, and any other count panics (toFixed: digits must be 0 to 100). NaN, the infinities and a number of magnitude 1e21 or more print as a template hole prints them, as JavaScript does. The answers are the same compiled and under Node.

asciiUpper(text) and asciiLower(text) change the letters a to z and A to Z and nothing else, so every other byte, the bytes of é and ß included, passes through. That is why they are not toUpperCase and toLowerCase, which JavaScript has and Nish does not: "ß".toUpperCase() is "SS", while asciiUpper("ß") is "ß". It is the case change an identifier wants, and it answers the same compiled and under Node.

padStart(text, width, fill) and padEnd(text, width, fill) repeat fill before or after text until the result is width long, cutting the last repeat at the width. A width no longer than text, or an empty fill, gives text unchanged. For ASCII text they are JavaScript's: padStart("7", 3, "0") is "007". The width counts what .length counts, UTF-8 bytes in a compiled program and UTF-16 units under Node, so on text that is not ASCII the two differ: padStart("é", 3, " ") is " é" compiled and " é" under Node. --warn-portability does not flag a pad call, because the length it reads is read inside the library and the result is a string (portability warnings), so pad only ASCII text, or work out the width with codePointLength. A fill that is not ASCII panics, because a cut at a byte width could split a character in two.

Together with nish/regex they turn a snake_case or kebab-case name into a camelCase identifier. A regex template cannot change case, so the regex only makes every separator an underscore, and the case helpers do the rest:

import { asciiLower, asciiUpper, split } from "nish/text";
import { Regex, regexCompile } from "nish/regex";

const toCamel = (separators: Regex, name: string): string => {
  const words = split(asciiLower(separators.replaceAll(name, "_")), "_");
  const camel: string[] = [words[0]];
  for (let i = 1; i < words.length; i++) {
    const word = words[i];
    camel.push(asciiUpper(word.substring(0, 1)));
    camel.push(word.substring(1));
  }
  return camel.join("");
};

export const main = (): i32 => {
  const compiled = regexCompile("[-\\s]+", "");
  if (compiled.isErr()) {
    console.log(compiled.error);
    return 1;
  }
  const separators: Regex = compiled.value;
  console.log(toCamel(separators, "user_account-ID"));
  console.log(toCamel(separators, "HTTP status-code"));
  console.log(toCamel(separators, "already__doubled_"));
  return 0;
};
$ nish run camel.ts
userAccountId
httpStatusCode
alreadyDoubled

indexOfAny(text, bytes, from) finds the first byte at or after from that is one of up to 16 ASCII characters, and is compiled to a vectorised search; it is not constant-time, so it is never for secret data (Byte search). firstDifference(left, right) answers the index of the first line where two arrays of lines differ.

codePointLength(text) counts Unicode code points, which is what a length limit on text a person typed usually means (JSON Schema's minLength and maxLength count them). .length does not: it counts UTF-8 bytes, so "é".length is 2 and "😀".length is 4, where codePointLength answers 1 for each. It answers the same when the program runs under Node, where .length counts UTF-16 units instead, for well-formed text, except text built from bytes: under Node, String.fromBytes, String.fromCharCode of a byte and a \u escape that nish/json decodes give one unit per byte, so such a string counts its UTF-8 bytes there.

nish/regex

Regular expressions in JavaScript's syntax. regexCompile(pattern, flags) answers a Result: the Regex, or the reason the pattern is refused.

import { Regex, RegexMatch, regexCompile } from "nish/regex";

export const main = (): i32 => {
  const compiled = regexCompile("(?<year>\\d{4})-(?<month>\\d\\d)", "");
  if (compiled.isErr()) {
    console.log(compiled.error);
    return 1;
  }
  const date: Regex = compiled.value;
  const m = new RegexMatch();
  if (date.exec("due 2026-10, paid 2026-11", 0, m)) {
    console.log(`${m.text()} at ${m.start()}, month ${m.named("month")}`);
  }
  console.log(date.findAll("due 2026-10, paid 2026-11").join(" "));
  console.log(date.replaceAll("2026-10", "$<month>/$<year>"));
  const bad = regexCompile("(a", "");
  if (bad.isErr()) {
    console.log(bad.error);
  }
  return 0;
};
$ nish run regex.ts
2026-10 at 4, month 10
2026-10 2026-11
10/2026
regex: missing ) at byte 0

A Regex has test, exec (into a RegexMatch you reuse), find, findAll, count, replace, replaceAll and split. A replacement template reads $&, $1 to $99, $<name>, $`, $' and $$, as in JavaScript. regexEscape(text) makes a pattern that matches text exactly. The flags are i, m, s and u, which changes nothing, as every pattern is read the u way; there is no g, since findAll and replaceAll are the global forms.

\p{…} matches a Unicode property, and \P{…} everything outside it, with JavaScript's names from Unicode 16.0: a General_Category value (\p{L}, \p{Lu}, \p{Uppercase_Letter}, \p{gc=Nd}), a script (\p{Script=Greek}, \p{sc=Grek}, or \p{scx=Deva} for Script_Extensions), or a binary property (\p{Alphabetic}, \p{Emoji}, \p{White_Space}, \p{Any}, \p{Assigned}). They work inside classes too, as in [\p{L}\p{Nd}_]+. A name JavaScript does not know is refused: regex: invalid property name at byte 0, or invalid property value for \p{sc=Lu}. A property escape cannot end a range in a class: [\p{L}-z] is refused (regex: invalid class range at byte 1), as it is in JavaScript.

The property and folding tables add about 80 KB to every program that imports nish/regex, whether or not it uses \p{…} or i: a pattern is read at run time, so the tables cannot be left out. Compiling a pattern whose i class or character folds a letter outside ASCII looks each such letter up in the folding table where it lies, through an index of every sixteenth entry, so nothing is decoded ahead of time: a single é adds about 0.2 µs to the compile. Other patterns never read the table.

Matching never backtracks, so a pattern cannot make one search slow: (a+)+$ against a long run of as takes as long as a+$. The price is that backreferences (\1) and lookaround ((?=…), (?<=…)) are refused when the pattern is compiled. So is a pattern past the engine's limits: a repeat count over 1,000, more than 1,024 capturing groups, more than 10,000 ranges in one class, more than 100,000 instructions once its repeats are written out, or property escapes and case folding that add more than 100,000 ranges to its classes.

Offsets are UTF-8 bytes, as s.length is, and \w, \d and \b are ASCII, as they are in JavaScript. The i flag folds every letter by Unicode's simple case folding, so é matches É and σ matches Σ and ς. One difference from JavaScript: no letter outside ASCII folds to one inside it, so the Kelvin sign K and the long ſ do not match k or s, nor [a-z], \p{Lu} or \p{Ll}.

Folding no letter outside ASCII into ASCII keeps an ASCII pattern under i ASCII: it folds A–Z to a–z with no table, and its search, its vectorised prefix scan included, never has to look for those two letters' three-byte and two-byte forms. Under i both still count as word characters to \w, \W, \b and \B, as they do in JavaScript with iu. Under i, \P{Lu} and [^\p{Lu}] differ, as they do in JavaScript: \P{Lu} is everything outside \p{Lu}, closed under folding, so it matches A through a, while [^\p{Lu}] negates the folded class [\p{Lu}], which holds every cased letter, so it matches neither A nor a.

Finding every match is one search per match, so findAll, count, replaceAll and split are quadratic when a long branch is preferred to a short one that wins: <[^>]*>|< over a run of <s reads the rest of the text at each. Bound the long branch (<[^<>]*>|<) or put the short one first.

A search runs a lazy DFA, a table of states built as the text needs them, and falls back to a slower engine only to fill in groups or when a pattern would need more states than the table holds (about 1 MiB). A Regex keeps that table and its other scratch memory between calls, so repeated searches allocate little beyond their answers — which also means one Regex is used by one thread at a time.

nish/json

jsonField(object, name) answers the value of one field of a flat JSON object, or null when the field is not there. It is a reader, not a parser: it does not validate, and a nested value is stepped over rather than searched. It reads exactly the shape nish --json prints, which is what it was written for (The CLI).

import { jsonField } from "nish/json";

export const main = (): i32 => {
  const line = `{"file":"p.ts","line":3,"code":"NL1047","message":"Loose equality is forbidden"}`;
  const code = jsonField(line, "code");
  if (code !== null) {
    console.log(`code ${code}`);
  }
  console.log(`${jsonField(line, "fix") === null}`);
  return 0;
};
$ nish run json.ts
code NL1047
true

A string value comes back unquoted; any other value comes back as written, so parseInt reads a number. jsonFields(object, names) reads several fields in one pass.

JsonWriter goes the other way: it writes any JSON value into a u8[] you own, placing every comma and colon itself, byte for byte what JSON.stringify writes. You say only what the value is:

import { JsonWriter } from "nish/json";

export const main = (): i32 => {
  const out = new Array<u8>(256);
  const w = new JsonWriter(out, 0);
  w.beginObject();
  w.key("message");
  w.string("Hello, \"World\"");
  w.key("count");
  w.i32(3);
  w.endObject();
  if (w.complete()) {
    console.log(w.text()); // {"message":"Hello, \"World\"","count":3}
  }
  return 0;
};

The JSON is out[0 .. w.end), and text() copies it into a string. A full buffer or a call out of order — a value where a key belongs, say — does not panic: it sets a negative JSON_WRITE_* code in w.failed, and every later call does nothing, so you write the whole value and check once at the end. jsonStringifyString(s) answers one string as a JSON string literal.

When your text is already bytes — a window of a request body, or what a JsonDocument holds — w.stringBytes(bytes, off, len) and w.keyBytes(bytes, off, len) write bytes[off .. off + len) exactly as string and key write a string of those bytes, without making the string. A window that is not inside bytes is JSON_WRITE_BAD_WINDOW, and nothing is written.

The writer has two modes, both off until you ask for them. Each is a setting of the writer rather than of one text: reset keeps it, so you set it once, and it stays until you set it again.

  • w.setIndent(n) writes what JSON.stringify(v, null, n) writes: one member or element a line, a space after each colon, and an empty {} or [] left on one line. n is clamped to 10, and 0 or a negative n is the minified text, the default. w.setIndentTab() indents with a tab. A raw fragment goes in as it is and is not re-indented.
  • w.setStrict(true) makes NaN and the infinities an error, JSON_WRITE_NON_FINITE (-8), with nothing written, where the default writes null as JSON.stringify does. A misuse before it, such as a value where a key is due, keeps its own code.
import { JSON_WRITE_NON_FINITE, JsonWriter } from "nish/json";

export const main = (): i32 => {
  const out = new Array<u8>(256);
  const w = new JsonWriter(out, 0);
  w.setIndent(2);
  w.beginObject();
  w.key("tags");
  w.beginArray();
  w.string("a");
  w.endArray();
  w.key("none");
  w.beginObject();
  w.endObject();
  w.endObject();
  console.log(w.text());
  w.setIndent(0);
  w.setStrict(true);
  w.reset(out, 0);
  w.f64(toF64(0) / toF64(0));
  console.log(`${w.failed === JSON_WRITE_NON_FINITE}`);
  return 0;
};
$ nish run pretty.ts
{
  "tags": [
    "a"
  ],
  "none": {}
}
true

jsonEncode from nish/json-typed writes through the writer you give it, so an indented or strict writer encodes a class indented or strictly too.

nish/json-tokenizer

jsonField trusts its input. When the JSON comes from someone else — a request body, a file a user hands you — read it with JsonTokenizer, which checks every byte against the JSON grammar and gives you the document one token at a time: an object or array opening or closing, a key, a string, a number, true, false or null. It reads a window of a u8[] and allocates nothing per token, so one tokenizer, reset for each new body, serves a whole connection.

import { JSON_TOK_END, JSON_TOK_KEY, JSON_TOK_NUMBER, JsonLimits, JsonTokenizer } from "nish/json-tokenizer";

const bytesOf = (text: string): u8[] => {
  const out: u8[] = [];
  const n: i32 = toI32(text.length);
  for (let i: i32 = 0; i < n; i++) {
    out.push(toU8(text.charCodeAt(i)));
  }
  return out;
};

export const main = (): i32 => {
  const body = bytesOf(`{"item":"tea","qty":3,"price":2.5}`);
  const t = new JsonTokenizer(body, 0, toI32(body.length), new JsonLimits());
  let total: f64 = 1;
  let k: i32 = t.next();
  while (k > 0) {
    if (k === JSON_TOK_KEY && (t.keyEquals("qty") || t.keyEquals("price"))) {
      if (t.next() === JSON_TOK_NUMBER) {
        total = total * t.numberF64();
      }
    }
    k = t.next();
  }
  if (k !== JSON_TOK_END) {
    console.log(`refused: code ${t.failed} at byte ${t.errorAt}`);
    return 1;
  }
  console.log(`${total}`); // 7.5
  return 0;
};

next() answers each token's kind, then JSON_TOK_END once the whole document has been read and nothing but whitespace follows it. A malformed document answers a negative JSON_TOK_ERR_* code instead; the code stays in failed, with the byte it was found at in errorAt, and every later call answers it again. Nothing in the input can make it panic or read outside the window you gave it.

  • Strings. stringInto(out, at) copies the current key or string into your array with its escapes decoded, and keyEquals(name) compares it without copying. Strings must be valid UTF-8, and a \u escape of half a surrogate pair on its own is refused, so what you copy out is always valid UTF-8.
  • Numbers. numberF64() reads the current number as JSON.parse would, correctly rounded. isInteger() and numberI64() read one that fits in an i64, and isU64() and numberU64() read one written in digits alone, with no minus sign, that fits in a u64, up to 18446744073709551615. Those go by how the number is written, so 1.0 and 1e2 are not integers to them. isIntegral() asks whether the value is whole however it is written: 1.0, 1e2, 100e-2 and 1e400 are, 1.5 and 10e-2 are not. That is JSON Schema's "integer". It reads the text, not a double, so it holds at any size and any exponent.
  • Skipping. skipValue() steps over whatever value comes next, however deeply it nests: use it for a field you do not want.
  • Limits. JsonLimits caps nesting depth (128), the bytes in one string (1 MiB), in one number (256) and in the whole input (16 MiB). Each limit has its own error code, and you can lower or raise each field before you make the tokenizer.
  • Duplicate keys. {"a":1,"a":2} is two keys. The tokenizer reports both, and what to do with the second is your decision.

Its security record is docs/security/json.md.

nish/json-typed

jsonDecode(d, out) reads one JSON object into an object of a class you declare, and jsonEncode(w, value) writes one back. There is nothing to register and no schema to write: the compiler writes the code from the class's fields at each call. A field can be an i32, u32, i64, u64, f64, boolean, string, an enum of either kind, another class, an array of any of those, a Map with string keys whose values are any of those, or a string, a class, an array or a map with | null; and a JsonNumber, which keeps a number's text exactly.

import { JsonWriter } from "nish/json";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDecoder, jsonDecode, jsonEncode } from "nish/json-typed";

class Player {
  name: string = "";
  score: i32 = 0;
}

class Team {
  players: Player[];
  captain: Player | null = null;

  constructor() {
    this.players = [];
  }
}

const bytesOf = (text: string): u8[] => {
  const out = new Array<u8>(toI32(text.length));
  for (let i: i32 = 0; i < toI32(out.length) && i < toI32(text.length); i += 1) {
    out[i] = toU8(text.charCodeAt(i));
  }
  return out;
};

export const main = (): i32 => {
  const body = bytesOf(`{"players":[{"name":"Ann","score":3}],"captain":null}`);
  const d = new JsonDecoder(body, 0, toI32(body.length), new JsonLimits());
  const team = new Team();
  const code = jsonDecode(d, team);
  if (code !== 0) {
    console.log(`error ${code} at byte ${d.errorAt}`);
    return 1;
  }
  const w = new JsonWriter(new Array<u8>(256), 0);
  jsonEncode(w, team);
  console.log(w.text()); // {"players":[{"name":"Ann","score":3}],"captain":null}
  return 0;
};

Decoding is strict. Every field must be there, each key may appear once, a number must fit its field (1.5 is not an i32), and null goes only into a field declared | null. A key the class has no field for is an error too, unless you set d.ignoreUnknown = true before decoding. Nothing panics: the first problem sets a negative code in d.failed and the byte it was found at in d.errorAt, and jsonDecode returns it.

The tokenizer's limits in JsonLimits — nesting depth, string, number and body size — apply as they would to any JSON you read. The depth limit is also how deep the decoder recurses, so keep it to a few thousand at most; and the writer stops at 64 levels, so an object decoded from deeper than that cannot be encoded back.

Decoding into the same object again reuses what it already holds: nested objects, arrays and any string that has not changed. A server can keep one decoder and one object per connection, call d.reset(body, 0, length) per request, and allocate nothing for a request that looks like the last. A class the decoder has to create — an array element past the end of the array, or a | null field that was null — is made with new C(), so its constructor takes no parameters. Encoding writes the fields in the order the class declares them, exactly as JSON.stringify would.

A u64 keeps every digit up to 2^64 - 1, which a JavaScript number cannot, and a negative number is a range error for it. A numeric enum is read and written as its number, and a string enum as its member's string. A value that is not one of the enum's members is a type error, JSON_DECODE_TYPE, and the field keeps the member it held. That makes a string enum field the way to take one of a fixed set of strings from input: the decoder checks it for you, and allocates nothing to do it.

import { JsonWriter } from "nish/json";
import { JsonLimits } from "nish/json-tokenizer";
import { JSON_DECODE_TYPE, JsonDecoder, jsonDecode, jsonEncode } from "nish/json-typed";

enum Role {
  Admin = "admin",
  Guest = "guest",
}

class Member {
  name: string = "";
  role: Role = Role.Guest;
}

const bytesOf = (text: string): u8[] => {
  const out = new Array<u8>(toI32(text.length));
  for (let i: i32 = 0; i < toI32(out.length) && i < toI32(text.length); i += 1) {
    out[i] = toU8(text.charCodeAt(i));
  }
  return out;
};

const read = (text: string, m: Member): i32 => {
  const body = bytesOf(text);
  const d = new JsonDecoder(body, 0, toI32(body.length), new JsonLimits());
  return jsonDecode(d, m);
};

export const main = (): i32 => {
  const m = new Member();
  if (read(`{"name":"Ann","role":"admin"}`, m) !== 0) {
    return 1;
  }
  console.log(m.role === Role.Admin ? "an admin" : "a guest"); // an admin
  const w = new JsonWriter(new Array<u8>(256), 0);
  jsonEncode(w, m);
  console.log(w.text()); // {"name":"Ann","role":"admin"}

  const bad = new Member();
  const code = read(`{"name":"Bo","role":"root"}`, bad);
  console.log(`${code === JSON_DECODE_TYPE} ${String(bad.role)}`); // true guest
  return 0;
};

A field can't be Role | null, because an enum is never nullable, and a Map can't be keyed by an enum; an array of one, Role[], is fine. A Map<string, V> is a JSON object whose keys you do not know in advance: decoding empties the map and fills it in the order the keys arrive, a key given twice is an error, and encoding writes the entries in the map's order.

import { JsonWriter } from "nish/json";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDecoder, jsonDecode, jsonEncode } from "nish/json-typed";

enum Level {
  Low = 0,
  High = 9,
}

class Inventory {
  serial: u64 = 0;
  level: Level;
  stock: Map<string, i32>;

  constructor() {
    this.level = Level.Low;
    this.stock = new Map<string, i32>();
  }
}

const bytesOf = (text: string): u8[] => {
  const out = new Array<u8>(toI32(text.length));
  for (let i: i32 = 0; i < toI32(out.length) && i < toI32(text.length); i += 1) {
    out[i] = toU8(text.charCodeAt(i));
  }
  return out;
};

export const main = (): i32 => {
  const body = bytesOf(`{"serial":18446744073709551615,"level":9,"stock":{"apples":3,"pears":0}}`);
  const inv = new Inventory();
  const code = jsonDecode(new JsonDecoder(body, 0, toI32(body.length), new JsonLimits()), inv);
  const w = new JsonWriter(new Array<u8>(256), 0);
  jsonEncode(w, inv);
  console.log(w.text()); // the same text back
  console.log(`${inv.stock.get("apples") ?? 0}`); // 3
  return code;
};

In a class of three or more fields, a key is matched to a field by its length first and then, where three or more field names share that length, by its byte at one position, before the whole key is compared, so a key is usually compared with one name rather than with each in turn.

All the maps of one decode take at most d.maxMapKeys keys between them, 4,096 unless you set it; a body with more is refused with JSON_DECODE_MAP_LIMIT rather than left to slow the maps down, and d.reset starts the count again. Set it to 0 (or below) and no map may take a key at all: {} is then the only map value accepted.

When the JSON's keys are not the names you want for your fields, or some keys may be left out, say so in one static readonly json object literal in the class. rename gives a field the key it has on the wire, and optional lists the fields whose key may be missing: a missing one sets the field back to its initializer, so an optional field needs one. Every other key is still required.

The literal is read by the compiler and is not a field. A name that is not one of the class's fields, two fields with the same key, or anything that is not written out as a literal, is a compile error.

A JsonNumber field keeps a number exactly as it was written, so a number wider than a u64 or with more digits than an f64 comes back byte for byte. Read it with text(), isI64() and i64(), isU64() and u64(), or f64(), and set it with setText.

import { JsonWriter } from "nish/json";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDecoder, JsonNumber, jsonDecode, jsonEncode } from "nish/json-typed";

class Payment {
  static readonly json = { rename: { userId: "user_id" }, optional: ["note"] };
  amount: JsonNumber;
  note: string = "";
  userId: i32 = 0;

  constructor() {
    this.amount = new JsonNumber();
  }
}

const bytesOf = (text: string): u8[] => {
  const out = new Array<u8>(toI32(text.length));
  for (let i: i32 = 0; i < toI32(out.length) && i < toI32(text.length); i += 1) {
    out[i] = toU8(text.charCodeAt(i));
  }
  return out;
};

export const main = (): i32 => {
  const body = bytesOf(`{"user_id":7,"amount":12345678901234567890.123456789}`);
  const p = new Payment();
  const code = jsonDecode(new JsonDecoder(body, 0, toI32(body.length), new JsonLimits()), p);
  const w = new JsonWriter(new Array<u8>(256), 0);
  jsonEncode(w, p);
  console.log(w.text()); // {"amount":12345678901234567890.123456789,"note":"","user_id":7}
  console.log(`${p.userId} ${p.amount.isU64()}`); // 7 false
  return code;
};

An optional field whose key was left out holds the same value as one whose key was given its initializer. When you need to tell them apart — to apply a partial update, say — presence names, for each field you want to know about, a boolean field of the same class. Every decoding sets it to whether the input gave that field's key, and a key given null counts as given.

A presence field has no key of its own: decoding never reads it from the input, encoding leaves it out, and no other option may name it. Recording a field allocates nothing, and a class that records none decodes exactly as fast as before. Read the presence fields only when jsonDecode answered JSON_DECODE_OK.

That is what a JSON Merge Patch (RFC 7396) needs. Make the patch a class of its own, with every field optional and recorded, then copy across the fields that were given. null on a | null field removes the member. null on a field that is not | null fails the decode with JSON_DECODE_TYPE, so you apply nothing.

An object field merges one level down by the same steps, and an array is replaced whole. A patch that is not an object ([1], "x") replaces the whole target in RFC 7396; here it fails the decode with JSON_DECODE_TYPE, and you apply nothing. Decode each patch into a fresh patch object: a decode refills arrays and maps in place, so a reused one shares them with the target. examples/merge-patch.ts is the whole recipe.

import { JsonLimits } from "nish/json-tokenizer";
import { JSON_DECODE_OK, JsonDecoder, jsonDecode } from "nish/json-typed";

class User {
  name: string = "";
  email: string | null = null;
}

class UserPatch {
  static readonly json = {
    optional: ["name", "email"],
    presence: { name: "hasName", email: "hasEmail" },
  };
  name: string = "";
  email: string | null = null;
  hasName: boolean = false;
  hasEmail: boolean = false;
}

const bytesOf = (text: string): u8[] => {
  const out = new Array<u8>(toI32(text.length));
  for (let i: i32 = 0; i < toI32(out.length) && i < toI32(text.length); i += 1) {
    out[i] = toU8(text.charCodeAt(i));
  }
  return out;
};

export const main = (): i32 => {
  const user = new User();
  user.name = "Ada";
  user.email = "ada@example.com";
  const body = bytesOf(`{"email":null}`);
  const patch = new UserPatch();
  const code = jsonDecode(new JsonDecoder(body, 0, toI32(body.length), new JsonLimits()), patch);
  if (code !== JSON_DECODE_OK) {
    return 1;
  }
  if (patch.hasName) {
    user.name = patch.name;
  }
  if (patch.hasEmail) {
    user.email = patch.email;
  }
  console.log(`${user.name} ${user.email === null}`); // Ada true
  return 0;
};

Tagged unions

A field, an array element or a whole document can be a discriminated union of classes. On the wire it is the variant's object with its tag key first, and the tag's value is the name of its enum member, or the member's string when the tags come from a string enum (Circle = "circle" writes "circle"); a variant's tag option writes another string (tag: "" is the same as no tag), and rename renames the tag key. Every variant must use the same key and a different string, and the compiler tells you where they clash.

jsonDecode fills the object you give it, but which variant a document holds is only known once its tag is read, so a union is decoded with jsonDecodeUnion(d, prev). It answers the object it decoded into: prev itself when the tag matches, otherwise a new variant. On a failure it answers prev, with the code in d.failed. That holds only at the top: a union field, array element or Map value inside the document may already hold the new variant, half read, so check d.failed before using anything it decoded. The tag may come anywhere in the object; when it is not the first key the decoder reads ahead to find it. A tag that is missing, not a string or no variant's is JSON_DECODE_TAG.

import { JsonWriter } from "nish/json";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDecoder, jsonDecodeUnion, jsonEncode } from "nish/json-typed";

enum Ev {
  Open,
  Close,
}

class Open {
  static readonly json = { rename: { kind: "type" }, tag: "open" };
  readonly kind: Ev.Open = Ev.Open;
  path: string = "";
}

class Close {
  static readonly json = { rename: { kind: "type" } };
  readonly kind: Ev.Close = Ev.Close;
  code: i32 = 0;
}

type Event = Open | Close;

const none = (): Event => new Close();

const bytesOf = (text: string): u8[] => {
  const out = new Array<u8>(toI32(text.length));
  for (let i: i32 = 0; i < toI32(out.length) && i < toI32(text.length); i += 1) {
    out[i] = toU8(text.charCodeAt(i));
  }
  return out;
};

export const main = (): i32 => {
  const body = bytesOf(`{"path":"/tmp/a","type":"open"}`);
  const d = new JsonDecoder(body, 0, toI32(body.length), new JsonLimits());
  const event = jsonDecodeUnion(d, none());
  if (d.failed !== 0) {
    return 1;
  }
  if (event.kind === Ev.Open) {
    console.log(event.path); // /tmp/a
  }
  const w = new JsonWriter(new Array<u8>(128), 0);
  jsonEncode(w, event);
  console.log(w.text()); // {"type":"open","path":"/tmp/a"}
  return 0;
};

A recursive union, such as an expression tree whose variants hold other expressions, works too: each union gets one decoding and one encoding function that every use calls. A Map<string, Expr> field holds a union as its values, and each value is decoded and encoded through that same function.

All three calls need the compiler: under Node, where the types are erased, they end the program with an error. The rules in full are in Typed JSON.

nish/json-value

nish/json-typed needs a class that says what the JSON looks like. When you do not know — a configuration file, a body you pass on after changing one field, a tool that prints whatever it is given — parse the document into a JsonDocument and walk it.

import { JsonLimits } from "nish/json-tokenizer";
import { JSON_VALUE_NUMBER, JsonDocument } from "nish/json-value";

export const main = (): i32 => {
  const doc = new JsonDocument(new JsonLimits());
  const root = doc.parseText(`{"id":12345678901234567890,"price":0.1000000000000000055511151231257827,"tags":["tea"]}`);
  if (root < 0) {
    console.log(`refused: code ${doc.failed} at byte ${doc.errorAt}`);
    return 1;
  }
  const id = doc.get(root, "id");
  console.log(`${doc.kind(id) === JSON_VALUE_NUMBER} ${doc.isI64(id)} ${doc.isU64(id)}`); // true false true
  console.log(`${doc.u64(id)}`); // 12345678901234567890
  console.log(`${doc.f64(doc.get(root, "price"))}`); // 0.1
  console.log(doc.string(doc.at(doc.get(root, "tags"), 0))); // tea

  doc.set(root, "tags", doc.newString("changed"));
  console.log(doc.stringify(root));
  // {"id":12345678901234567890,"price":0.1000000000000000055511151231257827,"tags":"changed"}
  return 0;
};

A value is a JsonValue, a handle into the document that made it, and every reader is a method of the document: kind, then boolean, string, numberText, length, at (an element, or an object's member by position), keyAt and get (a member by key). A handle that names nothing answers an empty value rather than panicking: get of a missing key is -1, and kind(-1) is JSON_VALUE_NONE.

Numbers are kept as they were written. This is where it differs from JavaScript. JSON.parse turns every number into a double, so the program above would see 12345678901234567000 and 0.1, and JSON.stringify would write those back. A JsonDocument keeps the text and reads it as what you ask for: isI64 and i64, isU64 and u64 for an integer that fits, f64 for the nearest double, and numberText for the digits themselves.

isIntegral asks whether the value is whole however it is written, so 1.0 and 1e2 are, though isI64 says no to both: they are not written as integers. Writing the document back writes the text, so an ID wider than any integer type, or a price with more digits than a double holds, comes out exactly as it came in.

Building. newNull, newBoolean, newString, newI64, newU64, newF64, newNumber (any JSON number, as text), newArray and newObject make values; push(array, v) appends and set(object, key, v) replaces a member in place or appends a new one. A value lives in at most one container, so adding one that is already in a container, or a container to itself, is refused with JSON_VALUE_ERR_ATTACHED and changes nothing.

Writing. stringify(v) answers minified text. write(v, w) writes into a JsonWriter you set up, so the buffer, its reuse and its error codes are the writer's, as in nish/json.

Errors and limits are the tokenizer's. parse(bytes, off, len) and parseText(s) answer the root, or a negative code: one of the nish/json-tokenizer codes, with the same JsonLimits and the byte in doc.errorAt. Nothing in the input can make it panic. The nesting depth is capped at 64, the writer's own limit, so every document a parse accepts can be written back.

A document holds at most doc.maxValues values, 262,144 unless you raise it; one more is JSON_VALUE_ERR_VALUES. Each value costs the document about 46 bytes (66 with ranges, below), kept for as long as the document is, so that cap, with the input's own size limit for the text the document copies, is what bounds a long-lived server's memory.

Duplicate keys are refused. {"a":1,"a":2} is JSON_VALUE_ERR_DUPLICATE, because two programs that read different members of one text can disagree about what it says. If you need to accept what JSON.parse accepts, set doc.duplicates = JSON_VALUE_DUPLICATE_LAST before parsing, and you get its answer: one member, at the place of the first, with the value of the last.

One document, many parses. Each parse replaces what the document held, and every handle from before it is gone: a handle kept from the last request reads as JSON_VALUE_NONE in this one, even when the two have the same shape.

The check that catches a stale handle counts in a cycle of 511 steps — one for each clear, one for each parse, two for a parse that fails — so do not keep handles across parses; it is there to catch the mistake. The document keeps its memory, so a server that keeps one document and parses each request into it stops allocating once it has seen its largest request. clear() empties it for building from scratch.

Where a value came from. parseRanged(bytes, off, len) and parseTextRanged(text) parse as parse and parseText do, and also record where each value and each key was in the input, so an error about a value can point at its line and column. start(v) and end(v) are the byte offsets of the value's first byte and one past its last, and keyStart(o, i) and keyEnd(o, i) the same for the key of member i of object o; a string's and a key's range includes its quotes.

The offsets are into the array you gave parseRanged, or into the string parseTextRanged read. Under Node, where parseTextRanged reads one byte per UTF-16 code unit, its offsets count code units, so past the first non-ASCII character they differ from a native build's, which count UTF-8 bytes.

import { JsonLimits } from "nish/json-tokenizer";
import { JsonDocument } from "nish/json-value";

export const main = (): i32 => {
  const doc = new JsonDocument(new JsonLimits());
  const text = `{"port": "eighty"}`;
  const root = doc.parseTextRanged(text);
  const port = doc.get(root, "port");
  console.log(`${doc.start(port)} ${doc.end(port)}`); // 9 17
  console.log(text.substring(doc.keyStart(root, 0), doc.keyEnd(root, 0))); // "port"
  return 0;
};

Ranges cost each value about 20 bytes more, so parse and parseText keep none, and after them all four answer -1. A value you build with the new* methods, and a key set adds, has no range either until you give it one with setRange(v, start, end) or setKeyRange(o, i, start, end), which is how a loader for another format that builds the tree itself can point back into its own text. set over an existing key keeps the key's range. Every parse and clear() drops the ranges the document held.

The two entry points are separate loops so that a program calling only parse runs the loop it always did. A program that links both, though, can see parse itself run slower, up to 16% on a number-heavy document, because the second loop changes how the first is laid out. A validator that wants ranges for its error messages calls parseRanged and nothing else, and one that does not calls parse; it should not mix the two to save the ranges' cost on the documents that turn out valid.

Objects keep their members in order, and get looks a key up by scanning them, so walk a large object once with keyAt and at rather than calling get for each of many keys. Under Node, where a string is UTF-16, parseText and the builders take one byte per character, so non-ASCII text differs from the native build there; parse over bytes reads the same both ways. Its security record is docs/security/json.md.

nish/json-number

A JSON number is text, and some texts say more than a double holds: 9007199254740993 and 9007199254740992 are one f64, and so are 0.1 and 0.10000000000000001. JSON Schema's minimum, maximum and multipleOf compare values, not doubles, and 0.3 is a multiple of 0.1 though 0.3 / 0.1 is 2.9999999999999996. This module reads the decimal digits themselves and never makes an f64.

import { JSON_NUMBER_ERR_GAP, jsonNumberCanonical, jsonNumberCompare, jsonNumberMultipleOf } from "nish/json-number";

export const main = (): i32 => {
  console.log(`${jsonNumberCompare("9007199254740993", "9007199254740992")}`); // 1
  console.log(`${jsonNumberCompare("1.0", "1e0")}`); // 0
  console.log(`${jsonNumberMultipleOf("0.3", "0.1")}`); // 1
  console.log(`${jsonNumberMultipleOf("4.5", "2")}`); // 0
  console.log(jsonNumberCanonical("0.0150e3")); // 15e0
  console.log(jsonNumberCanonical("1.50")); // 15e-1
  console.log(`${jsonNumberMultipleOf("1e1000000000", "7") === JSON_NUMBER_ERR_GAP}`); // true
  return 0;
};

jsonNumberCompare(a, b) answers -1, 0 or 1, and jsonNumberMultipleOf(value, divisor) answers 1 or 0 (signs do not matter, and zero is a multiple of anything). jsonNumberCanonical(text) is the shortest spelling of the value, digits without trailing zeros and an exponent (15e-1 for 1.50, 15e0 for 0.0150e3, 0 for -0), so two texts are equal values exactly when their canonical texts are the same string.

Each of them refuses rather than guessing, jsonNumberCompare and jsonNumberMultipleOf with a negative code and jsonNumberCanonical with "", and the first argument's refusal comes first:

  • JSON_NUMBER_ERR_SYNTAX: the text is not one JSON number (01, 1., +1, a blank around it).
  • JSON_NUMBER_ERR_EXPONENT: the exponent is past JSON_NUMBER_MAX_EXPONENT, 10^15 either way, and a digit is not zero (0e9999999999999999 is zero).
  • JSON_NUMBER_ERR_DIVISOR: multipleOf by zero.
  • JSON_NUMBER_ERR_GAP: multipleOf would take more than JSON_NUMBER_MAX_STEPS (2^20) digit steps. 1e1000000000 is twelve bytes of input and a billion digits of dividend, so the bound is what keeps a hostile schema or document from costing a billion steps.

A comparison costs the digits of the two texts; multipleOf costs the value's digits from its first to the divisor's last place, times the divisor's digits past eighteen, and never more than the bound.

nish/json-equal

JSON Schema's const, enum and uniqueItems ask whether two JSON values are equal, and its rule is not JavaScript's: numbers are equal by value, so 1 and 1.0 are, objects are equal whatever the order of their members, and true is not 1.

import { jsonEqual, jsonEqualCanonical, jsonEqualFirstDuplicate } from "nish/json-equal";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDocument } from "nish/json-value";

export const main = (): i32 => {
  const da = new JsonDocument(new JsonLimits());
  const db = new JsonDocument(new JsonLimits());
  const a = da.parseText(`{"id":1,"tags":["x"]}`);
  const b = db.parseText(`{"tags":["x"],"id":1.0}`);
  console.log(`${jsonEqual(da, a, db, b)}`); // true
  console.log(jsonEqualCanonical(da, a)); // o2:2:idn1e0;4:tagsa1:s1:x
  const list = da.parseText(`[1, true, "1", 1e0]`);
  console.log(`${jsonEqualFirstDuplicate(da, list)}`); // 3
  return 0;
};

jsonEqual(da, a, db, b) compares a value of one document with a value of another, or of the same one (a document's handles last until it parses again, so two parsed texts take two documents); a handle that names no value is equal to nothing. jsonEqualFirstDuplicate(doc, array) is the position of the first element equal to an earlier one, -1 when there is none, so uniqueItems holds exactly when it is -1.

jsonEqualCanonical is a text that is the same for exactly the values jsonEqual calls equal, which makes it a key for a Map or Set of JSON values; it is not JSON. jsonEqualHash is a 64-bit FNV-1a hash of it. The hash is not keyed, so anyone can make values that collide: use the canonical text, not the hash, as a table's key when the values come from someone else.

A number past the exponent bound of nish/json-number is equal only to the same text. None of the functions recurses, so a document as deep as its limits allow compares. jsonEqual costs the values' sizes, plus sorting each pair of objects' keys; the canonical text and the first duplicate cost the canonical text's length.

nish/json-pointer

A JSON Pointer (RFC 6901) names one value inside a document: /tags/0 is the first element of the member tags. A token is a member's key, with ~1 for a / in it and ~0 for a ~, or an array index written without a leading zero.

import { JSON_POINTER_NONE, isRelativeJsonPointer, jsonPointerEscape, jsonPointerParse, jsonPointerResolve } from "nish/json-pointer";
import { JsonLimits } from "nish/json-tokenizer";
import { JsonDocument } from "nish/json-value";

export const main = (): i32 => {
  const doc = new JsonDocument(new JsonLimits());
  const root = doc.parseText(`{"a/b":[10,20],"tags":["tea"]}`);
  console.log(`${doc.i64(jsonPointerResolve(doc, root, "/a~1b/1"))}`); // 20
  console.log(`${jsonPointerResolve(doc, root, "/tags/1") === JSON_POINTER_NONE}`); // true
  console.log(`/${jsonPointerEscape("a/b")}/0`); // /a~1b/0
  const tokens = jsonPointerParse("/a~1b//~0");
  console.log(tokens === null ? "refused" : tokens.join("|")); // a/b||~
  console.log(`${isRelativeJsonPointer("1/tags")}`); // true
  return 0;
};

jsonPointerResolve(doc, root, pointer) answers the value, root itself for the empty pointer, JSON_POINTER_NONE when the pointer names nothing there (a missing key, an index past the end, -, a token on a number) and JSON_POINTER_ERR_SYNTAX when the text is not a pointer. jsonPointerParse answers the decoded tokens, or null; jsonPointerEscape writes one token. The pointer is the plain string, not a URI fragment: #/a is not one.

isJsonPointer and isRelativeJsonPointer check the form JSON Schema's json-pointer and relative-json-pointer formats name, reading the text once and allocating nothing. A relative pointer (0#, 1/tags) is checked but not resolved: what it is relative to is the validator's to know. A resolve costs the pointer's length, and each member step the size of the object it looks in.

nish/json-format

A JSON Schema names a string's shape with format: "date-time", "email", "ipv6", "uri". nish/json-format checks the formats of draft 2020-12, one predicate each, and a dispatcher for a format named in a schema.

import { JSON_FORMAT_UNKNOWN, JSON_FORMAT_VALID, isEmail, isIpv6, jsonFormatCheck, jsonFormatId, jsonFormatValidate } from "nish/json-format";

export const main = (): i32 => {
  console.log(`${isEmail("joe.bloggs@example.com")} ${isIpv6("::ffff:192.168.0.1")}`); // true true

  const id = jsonFormatId("date-time"); // once, when the schema is read
  console.log(`${jsonFormatCheck(id, "1963-06-19T08:30:06Z") === JSON_FORMAT_VALID}`); // true
  console.log(`${jsonFormatCheck(id, "1963-06-19 08:30:06Z")}`); // 0, invalid
  console.log(`${jsonFormatValidate("idn-email", "x@y") === JSON_FORMAT_UNKNOWN}`); // true
  return 0;
};

The predicates are isDateTime, isDate, isTime, isDuration, isEmail, isHostname, isIpv4, isIpv6, isUuid, isUri, isUriReference, isIri, isIriReference, isUriTemplate and isRegex. Each is the grammar of the RFC the specification names, applied to the whole string, and each but isRegex reads the string once or twice and allocates nothing.

The dispatcher answers JSON_FORMAT_VALID (1), JSON_FORMAT_INVALID (0) or JSON_FORMAT_UNKNOWN (-1). Unknown is the answer for idn-email, idn-hostname, json-pointer, relative-json-pointer and any other name, and a validator accepts what it cannot check. jsonFormatId compares the name with each format in turn, so resolve it once per schema and call jsonFormatCheck per instance; jsonFormatValidate(name, value) does both at once.

Some answers are stricter than you may expect:

  • duration follows RFC 3339's Appendix A grammar as written, so its units come in order and none is skipped: P1Y2M and PT1H2M are valid, and P1Y1D and PT1H1S are not.
  • hostname does not decode an xn-- label's Punycode.
  • regex is a pattern nish/regex compiles. It refuses lookaround and backreferences, and a lone { or ], all of which ECMA-262 accepts, so isRegex("(?=a)a") is false.

uri and its relatives are checked against RFC 3986 and RFC 3987 here rather than by nish/net/url, which refuses valid URIs that could mislead a client, such as a percent-encoded host. Every case of JSON-Schema-Test-Suite's optional/format for these formats gives the suite's answer (tests/link/std_json_format).

nish/yaml

parseYaml(text) reads a YAML 1.2 stream — a configuration file, a CI workflow, a Kubernetes manifest — and answers a YamlStream: every document, every node, and every problem found, each with the byte range of the source it is about. A parse never fails and never panics on its input; a stream that is not valid YAML still answers its nodes, as the parser's best reading of it, beside the errors that say why.

import { YAML_INT, YAML_MAP, keyText, parseYaml, pathNode, yamlErrorName } from "nish/yaml";
import { YamlLineIndex } from "nish/yaml-lines";

export const main = (): i32 => {
  const text = "name: tea\nport: 8080\ntags: [green, black]\n";
  const stream = parseYaml(text);
  const root = stream.documentRoot(0);
  if (stream.kind(root) === YAML_MAP) {
    let i: i32 = 0;
    while (i < stream.count(root)) {
      const pair = stream.child(root, i);
      console.log(`${keyText(stream, stream.key(pair))} = ${keyText(stream, stream.value(pair))}`);
      i += 1;
    }
  }
  // name = tea
  // port = 8080
  // tags = [ green, black ]

  const port = pathNode(stream, root, ["port"]);
  console.log(`${stream.type(port) === YAML_INT} ${stream.intValue(port)}`); // true 8080
  console.log(stream.text(pathNode(stream, root, ["tags", "1"]))); // black
  console.log(`bytes ${stream.start(port)}..${stream.end(port)}`); // bytes 16..20

  const bad = parseYaml("a: 1\nb: [1,,2]\na: 3\n");
  const lines = new YamlLineIndex(bad.source);
  let d: i32 = 0;
  while (d < bad.diagnosticCount()) {
    const at = bad.diagnosticStart(d);
    console.log(`${lines.line(at)}:${lines.column(at)} ${yamlErrorName(bad.diagnosticCode(d))} ${bad.diagnosticMessage(d)}`);
    d += 1;
  }
  // 2:7 UNEXPECTED_COMMA Missing value before ","
  // 3:1 DUPLICATE_KEY Map key "a" is duplicated
  return bad.errorCount() === 2 ? 0 : 1;
};

A node is an i32 handle into the stream, and -1 is no node. Its kind is YAML_SCALAR, YAML_MAP, YAML_SEQ, YAML_ALIAS or YAML_PAIR. count(n) and child(n, i) walk a collection: a map's children are its pairs, in the order they were written, and a pair's key and value are nodes; a sequence's children are its items. An empty value (a:) is no node, so its value is -1 and reads as null. An alias's target is the node its anchor named, and resolved(n) follows an alias to it. anchor, tag and style answer what was written; a !! tag is kept without its prefix ("int").

Scalars are read when you ask. text(n) decodes a scalar — quotes, escapes, folding and block scalars — and type(n) resolves it against the core schema: YAML_NULL, YAML_BOOL, YAML_INT, YAML_FLOAT or YAML_STRING, with boolValue, intValue and floatValue to read it. YAML 1.1's yes and on are strings, as 1.2 says. Nothing is decoded until it is read, so a program that wants one key pays for one key.

Paths. pathNode(stream, root, path) follows a path of map keys and sequence indices ("0", never "01") through aliases, and answers the node the last matching step reached; pathMatch says how many steps matched and pathPair answers the last pair it stepped through, whose key is where a message about that setting should point.

A key is matched by keyText, the text it was written as, so 1 and 01 are different keys and a null key (~, null) is matched by "". When a map repeats a key, the last one wins. A step onto an empty value (a:) matches, so pathMatch counts it and pathPair names its pair, but it reaches no node: pathNode answers -1.

Merge keys. A path follows << merge keys (a plain <<, untagged or !!merge): a key a map does not hold is looked up in the maps its << keys name — a map, an alias to one, or a sequence of those or an alias to one — in the order they are written.

The map's own keys win, then the first merged map that has the key, and the node and pair found are the ones inside that map, so a message points at the line that set it. A recursive alias ends the search. A << whose value names no map is an error, BAD_MERGE. The nodes themselves are not changed: a map still holds the pairs it was written with.

import { parseYaml, pathNode, pathPair } from "nish/yaml";
import { YamlLineIndex } from "nish/yaml-lines";

export const main = (): i32 => {
  const text = "defaults: &defaults\n  adapter: postgres\n  host: localhost\n\ndevelopment:\n  <<: *defaults\n  host: dev.local\n";
  const stream = parseYaml(text);
  const root = stream.documentRoot(0);
  const lines = new YamlLineIndex(text);
  const adapter = pathNode(stream, root, ["development", "adapter"]);
  console.log(`${stream.text(adapter)} at ${lines.line(stream.start(adapter))}:${lines.column(stream.start(adapter))}`); // postgres at 2:12
  console.log(stream.text(pathNode(stream, root, ["development", "host"]))); // dev.local
  const pair = pathPair(stream, root, ["development", "adapter"]);
  return pair === pathPair(stream, root, ["defaults", "adapter"]) ? 0 : 1;
};

A read never panics. A handle that names no node, or a read that does not fit the node — count of a scalar, intValue of a string — answers a default: -1 for a node or an offset, and 0, false or "" otherwise.

Ranges and lines. start(n) and end(n) are byte offsets into stream.source, as are a diagnostic's diagnosticStart and diagnosticEnd. new YamlLineIndex(text) from nish/yaml-lines turns an offset into a 1-based line and column (in characters; byteColumn counts bytes); a line break is LF, CR LF or a lone CR.

Diagnostics. diagnosticCount, errorCount and warningCount count them; each has a YAML_ERR_* code, named by yamlErrorName the way other YAML tools name it, diagnosticIsWarning, a message and its range, sorted by where it starts. An error means the text is not valid YAML; a warning means it is valid and probably not what was meant, such as !!int x. A map that repeats a key is an error, DUPLICATE_KEY.

Options. parseYamlWith(text, options) takes a YamlOptions: maxDepth (how deeply collections nest before DEPTH_LIMIT, 1000 by default; a value outside 1 to 1000 is a mistake in the program, so parseYamlWith panics on it), uniqueKeys, merge (whether a plain << key is a merge key: exempt from the duplicate check, checked for BAD_MERGE and followed by paths; on by default), keepComments (keep every comment's range, for commentCount and comment) and firstDocumentOnly (warn MULTIPLE_DOCUMENTS and stop after the first). Tabs are never indentation, as YAML says.

Loading YAML into a JsonDocument

new YamlJsonLoader() from nish/yaml-json loads one document of a YamlStream into a nish/json-value JsonDocument, so a program that reads JSON reads YAML with the same code. load(stream, d, doc) clears doc and answers the root, with every value's range (start, end) and every key's (keyStart, keyEnd) set to the YAML node's bytes.

A scalar is typed by the core schema: 0x1F and 0o17 load as the exact decimal numbers 31 and 15, and .inf, -.inf and .nan, which JSON cannot hold, are refused. An alias loads as a copy of its anchor's value, a << merge key adds the merged maps' keys the map does not have itself, and a key that is not a string loads as its source text (1, [ a, b ]).

Because the ranges are byte offsets into the text either way, one YamlLineIndex turns them into line:column whether the document came from JSON or from YAML:

import { JsonLimits } from "nish/json-tokenizer";
import { JsonDocument } from "nish/json-value";
import { parseYaml } from "nish/yaml";
import { YamlJsonLoader } from "nish/yaml-json";
import { YamlLineIndex } from "nish/yaml-lines";

const place = (text: string, doc: JsonDocument, root: i32): string => {
  const lines = new YamlLineIndex(text);
  const at = doc.start(doc.get(root, "port"));
  return `${lines.line(at)}:${lines.column(at)}`;
};

export const main = (): i32 => {
  const json = `{\n  "name": "api",\n  "port": 8080\n}`;
  const yaml = "name: api\nport: 8080\n";
  const fromJson = new JsonDocument(new JsonLimits());
  const fromYaml = new JsonDocument(new JsonLimits());
  const loader = new YamlJsonLoader();
  const a = fromJson.parseTextRanged(json);
  const b = loader.load(parseYaml(yaml), 0, fromYaml);
  console.log(`json ${place(json, fromJson, a)}, yaml ${place(yaml, fromYaml, b)}`);
  return b < 0 ? 1 : 0;
};

It prints json 3:11, yaml 2:7.

A load fails whole. A negative answer is a code, also in failed, with the range it is about in errorStart and errorEnd, and the document is left empty; yamlJsonErrorName names it. A document with a YAML error is YAML_JSON_ERR_YAML, a non-finite number YAML_JSON_ERR_NOT_FINITE, and nesting past maxDepth (64) YAML_JSON_ERR_DEPTH. An alias bomb is refused before anything is built: the values the aliases expand to are counted first, and past the document's maxValues the load is YAML_JSON_ERR_EXPANSION at the alias; copies whose text passes the document's JsonLimits.maxTotalBytes are refused the same way.

Writing YAML

stringifyYaml(doc, v) from nish/yaml-write writes a nish/json-value value as YAML that parseYaml reads back as the same tree: block style with a two-space indent, {} and [] for an empty object and array, null, true and false plain, a number as its JSON text, and a trailing newline. It answers "" for a handle that names no value and for a tree deeper than JSON_WRITER_MAX_DEPTH, the limit stringify keeps.

import { JsonLimits } from "nish/json-tokenizer";
import { JsonDocument } from "nish/json-value";
import { parseYaml } from "nish/yaml";
import { stringifyYaml } from "nish/yaml-write";

export const main = (): i32 => {
  const doc = new JsonDocument(new JsonLimits());
  const root = doc.parseText(`{"name":"nish","version":"0.24","tags":["yaml","no"],"empty":{}}`);
  const yaml = stringifyYaml(doc, root);
  console.log(yaml);
  return parseYaml(yaml).errorCount();
};

It prints:

name: nish
version: "0.24"
tags:
  - yaml
  - "no"
empty: {}

Which strings are quoted. A string is written plain unless YAML would read it as something else, and then double-quoted:

  • an empty string;
  • ~ and the words null, true, false, yes, no, on, off, y and n in any case;
  • anything that starts like a number (an optional sign, then a digit or . and a digit) or is .inf or .nan, signed or not, in any case;
  • a string that starts with an indicator (@, `, \, *, &, !, |, >, %, ', ", #, ,, [, ], {, }), with -, ? or : followed by a space or nothing, or with --- or ...;
  • a string with a leading or trailing space, a trailing :, or a : or # inside;
  • a key <<, which would be a merge key;
  • and a string holding a control character, a line break, U+2028, U+2029, U+FEFF, U+FFFE or U+FFFF, which are escaped.

A key longer than 1024 bytes as written is an explicit key, ? key and then : value.

nish/pair

Pair<A, B> is an interface with two fields, first and second, and nothing else. It is how a function answers two values: you return an object literal and the annotation gives it its type.

import { Pair } from "nish/pair";

const minMax = (xs: i32[]): Pair<i32, i32> => {
  let lo = xs[0];
  let hi = xs[0];
  for (const x of xs) {
    if (x < lo) {
      lo = x;
    }
    if (x > hi) {
      hi = x;
    }
  }
  return { first: lo, second: hi };
};

export const main = (): i32 => {
  const r = minMax([4, -2, 9, 3]);
  console.log(`${r.first} ${r.second}`); // -2 9
  return 0;
};

Import it with a plain import: import type is not supported. Use Pair to return two values, not to store them side by side — a class with named fields says more.

nish/map

Two helpers for the global Map. getOrInsert(m, key, value) answers the value at key, first storing value there if the key was missing, in one lookup. reserve(m, n) makes room for n entries up front, so filling the map never has to grow it.

import { getOrInsert, reserve } from "nish/map";

export const main = (): i32 => {
  const counts = new Map<string, i32>();
  reserve(counts, 8);
  for (const word of ["a", "b", "a", "c", "a"]) {
    counts.set(word, getOrInsert(counts, word, 0) + 1);
  }
  for (const word of counts.keys()) {
    const n = counts.get(word);
    if (n !== undefined) {
      console.log(`${word} ${n}`);
    }
  }
  return 0;
};
$ nish run words.ts
a 3
b 1
c 1

Map itself is described in Maps and sets.

nish/random

new Rng(seed) makes a generator of pseudo-random numbers from an i32 seed: the same seed always answers the same sequence, natively and under Node. nextInt(lo, hi) answers a whole number from lo to hi, both included, each with equal chance; nextI32() answers the next 32 bits as an i32, and nextF64() a number in [0, 1). Every seed works, 0 included, and nearby seeds answer unrelated sequences, so a task or a parallel body can seed its own from its index. It is not for keys or passwords: those come from crypto.getRandomValues.

import { Rng } from "nish/random";

export const main = (): i32 => {
  const r = new Rng(42);
  const rolls: string[] = [];
  for (let i: i32 = 0; i < 5; i++) {
    rolls.push(`${r.nextInt(1, 6)}`);
  }
  console.log(rolls.join(" ")); // 1 5 6 6 4, every run
  return 0;
};

nextInt(6, 1), with lo above hi, stops the program with Rng.nextInt: lo is greater than hi.

import { Rng } from "nish/random";

export const main = (): i32 => {
  const r = new Rng(2024);
  let inside: i32 = 0;
  for (let i: i32 = 0; i < 1000; i++) {
    const x = r.nextF64();
    const y = r.nextF64();
    if (x * x + y * y < 1.0) {
      inside = inside + 1;
    }
  }
  console.log(`${inside > 700 && inside < 870}`); // true: about 785 of 1000
  return 0;
};

A generator is an ordinary object you step, so a task or a parallel body that makes its own may use it freely; one shared between tasks would be a write they all see, and is refused. See Random numbers in the reference.

nish/cli

nish/cli builds a command-line tool's --help: a header, usage lines, sections of flags with their placeholders and descriptions lined up in columns, and notes. It is what nish --help itself is built with.

import { HelpMenu, agentDriven, styleFor } from "nish/cli";

export const main = (): i32 => {
  const menu = new HelpMenu(styleFor(1, agentDriven()), "greet", 24, 30);
  menu.header("1.2.0", "say hello");
  menu.usage("<name> [flags]", "greet someone");
  menu.section("Flags");
  menu.row("--shout", "", "in capitals");
  menu.row("--times", "<n>", "say it n times");
  console.log(menu.text());
  return 0;
};

Piped or redirected, that prints:

greet 1.2.0  say hello

usage: greet <name> [flags]   greet someone

Flags
  --shout               in capitals
  --times <n>           say it n times

A HelpMenu is made with a Style, the program's name, the column a row's description starts at and the column a usage line's starts at. Add lines in order: header(version, summary), usage(shape, description) (the first one says usage:, the rest line up under it), section(title), row(flag, placeholder, description), note(text) for a line of prose and value(text) for a path or value to copy.

title(version, summary) is header without its blank line after, and line(text) adds a line exactly as given, so a menu can put a rule or a banner under its title (paint it with the same Style, and keep it to the menu's width). An empty placeholder or description is left out. A flag too long for its column keeps two spaces before its description. text() answers the whole menu. Columns are counted in bytes, so they line up for ASCII text.

The colours are your terminal's own. A Style paints headings in bold magenta, flags in cyan, placeholders in yellow and descriptions faint, using the basic ANSI colours your terminal theme defines, never a fixed shade, so the menu takes on whatever palette the reader chose. A terminal that declares no colours (tput colors below 8, as for TERM=vt100) gets the same structure in bold and underline instead; forced colour is always the theme's.

styleFor(fd, agent) decides for descriptor fd (1 for stdout, 2 for stderr) whether to paint at all, the same way nish --help does: colour on a terminal, plain text in a pipe or a file, NO_COLOR always off, FORCE_COLOR or CLICOLOR_FORCE on even when piped, and TERM=dumb or CLICOLOR=0 off (the CLI page has the full table).

agentDriven() says whether a coding agent is running the program: NISH_AGENT decides when it is set, a no (0, false, no, off) for no and anything else for yes, and otherwise CLAUDECODE or GEMINI_CLI set to anything but empty or a no means yes. With agent true styleFor is always plain: the escapes would only cost the agent tokens.

HelpMenu and Style reach no capability. styleFor and agentDriven reach env, and styleFor reaches process.spawn too: unless colour is forced or refused first, it asks a shell whether the descriptor is a terminal and how many colours it has. To keep a program free of both, make the style yourself: new Style(on, colour), where on turns painting on and colour picks the theme's colours over the monochrome failsafe.

Colours

A Style also paints any text, not only a menu, with the same policy: plain when on is false, and the theme's colours only when colour is true.

group methods under the failsafe
foreground black red green yellow blue magenta cyan white, and brightBlack (gray) through brightWhite plain
background bgBlack through bgWhite, and bgBrightBlack through bgBrightWhite plain
attributes bold faint italic underline inverse strikethrough painted
raw sgr(text, codes), any SGR parameter list: "1;31", "38;5;208", "38;2;255;128;0" plain

Each answers text between its escape and a reset, or text unchanged when it paints nothing. The sixteen colours are the ones your terminal theme redefines; a 256-colour or truecolor tone ignores the theme, so it goes through sgr, where you ask for it on purpose. The SGR_ constants (SGR_RED, SGR_BRIGHT_BLACK, SGR_BG_BLUE, SGR_BOLD, ...) spell the codes, so s.sgr(text, `${SGR_BOLD};${SGR_RED}`) combines them without numbers. Nesting works too, but every painter ends with a full reset, so text after an inner painted part loses the outer marks: join the codes for anything longer than one part.

The menu's roles can be recoloured. Each role a HelpMenu paints keeps two SGR lists, one for the theme and one for the failsafe: headingThemed/headingMono, nameThemed/nameMono, flagThemed/flagMono, metaThemed/metaMono and dimThemed/dimMono. Set a pair before you build the menu; an empty list leaves that role plain. The defaults are the ones described above.

import { HelpMenu, SGR_BOLD, SGR_GREEN, agentDriven, styleFor } from "nish/cli";

export const main = (): i32 => {
  const style = styleFor(1, agentDriven());
  style.flagThemed = `${SGR_BOLD};${SGR_GREEN}`;
  const menu = new HelpMenu(style, "greet", 24, 30);
  menu.header("1.2.0", "say hello");
  menu.row("--shout", "", "in capitals");
  console.log(menu.text());
  console.log(`${style.red("error:")} ${style.bold("no name given")}`);
  return 0;
};

Parsing flags

nish/cli-flags reads a command line against a table of flags you declare once. The same table writes the help rows, so the two cannot drift apart.

import { HelpMenu, Style } from "nish/cli";
import { FlagTable } from "nish/cli-flags";

export const main = (): i32 => {
  const flags = new FlagTable("git-lite");
  flags.command("remote add", "<name> <url>", "add a remote");
  flags.string("output", "<file>", "where to write");
  flags.number("jobs", "<n>", "parallel jobs");
  flags.boolean("verbose", "say more");
  flags.repeatable("include", "<dir>", "search here too");
  const got = flags.parseFrom(process.argv, 1);
  if (!got.ok) {
    got.exitUsage();
  }
  if (got.command === "") {
    const menu = new HelpMenu(new Style(false, true), "git-lite", 28, 30);
    menu.usage("<command> [flags]", "a small git");
    flags.writeHelp(menu);
    console.log(menu.text());
    return 0;
  }
  console.log(`${got.command}: ${got.positionals.join(" ")} jobs ${got.num("jobs")}`);
  return 0;
};

git-lite --jobs 4 remote add origin https://example.org --include a --include b runs remote add with positionals origin and the URL, jobs 4 and include a then b. With no arguments it prints:

usage: git-lite <command> [flags]  a small git

Commands
  remote add <name> <url>   add a remote

Flags
  --output <file>           where to write
  --jobs <n>                parallel jobs
  --verbose                 say more
  --include <dir>           search here too

Declare each flag by its name without the dashes, a placeholder for the help and a description: string, number and repeatable take a value, boolean does not. command(path, shape, description) declares a subcommand of one or more words, with what follows it in the help.

A name declared twice panics, because the table is your program's own text, and so does one no command line could match: a flag name with a leading -, an = or whitespace, a command word with a leading -, or a flag no-x beside a boolean x, whose --no-x it would be. parseFrom(process.argv, 1) reads the command line after the program's own path, and parse(args) reads every string in args.

The grammar is the long-option one most tools share, without short options:

  • --name value and --name=value give a value; --name= gives the empty string. The next argument is the value whatever it looks like, so --jobs -1 is minus one and --output -- writes to a file named --.
  • --name sets a boolean and --no-name clears it. A boolean takes no =value.
  • A repeatable flag keeps every value in the order given. Any other flag given twice is an error.
  • -- ends the flags: everything after it is a positional, even --verbose and a second --. A positional that starts with -, such as a negative number, needs a -- before it, or it reads as a flag. A lone - is an ordinary word, as for a tool that reads - as stdin.
  • The words that are not flags may come before, between or after them. The leading ones choose the command, the longest declared path they spell, and the rest are positionals. A flag belongs to the whole table, not to one command: check got.command to refuse one where it does not apply.

Reading the answer. got.command is the matched path, or "" when no words were given. str, num, bool and all read a flag of that kind by name; a flag that was not given reads as "", 0, false or no values, and has(name) tells that apart from one given as such.

A number is a finite f64 written in decimal: an optional sign, digits with an optional fraction, and an optional exponent (-2.5, .5, 1e3), in ASCII. Hex and binary, Infinity, an empty value, spaces round it, and a value too large for an f64 (1e999) are refused, the same natively as under Node. Asking for a name the table does not declare, or as the wrong kind, panics.

A usage error is data. parse stops at the first one: an unknown flag (any -x with one dash is one), a flag missing its value, a number that is not one, a flag given twice, a value on a boolean, or words that spell no declared command once any is declared. got.ok is false, and got.code (a FLAG_ERR_* constant), got.message and got.argument say what went wrong, so a program can word it its own way. got.exitUsage() is the usual ending: it writes

git-lite: --jobs needs a number, not "four"
usage: git-lite remote add <name> <url> [flags]

to stderr and exits 2, the status nish itself uses for a usage error. usageError() answers those two lines without printing them. exitUsage reaches exit; the rest of the module reaches no capability.

nish/cli-args

nish/cli-args parses the command line of a tool with many commands (api users keys list 42 --limit 5): you declare the tree once, parse walks the arguments, and the answer is the id of the command they reached and every flag's value with where it came from. finish then prints the help, the version or an error, the way nish/cli lays help out.

import { Cli, ROOT, finish } from "nish/cli-args";
import { agentDriven, styleFor } from "nish/cli";

const LIST: i32 = 1;

export const main = (): i32 => {
  const cli = new Cli("api", "1.4.0", "Talk to the Example API");
  const raw = cli.globalFlag("-r, --raw-output", "print JSON as it arrives");
  const users = cli.group(ROOT, "users", "Manage users");
  const list = cli.command(users, "list", LIST, "List users");
  const limit = cli.flag(list, "-l, --limit <n>", "at most n users");
  cli.defaultValue(limit, "20");

  const r = cli.parse(process.argv, 1);
  const agent = agentDriven();
  finish(r, styleFor(1, agent), styleFor(2, agent));
  if (r.id === LIST) {
    console.log(`listing ${r.value(limit) ?? ""} users, raw: ${r.flag(raw)}`);
  }
  return 0;
};

api users list -l 5 -r prints listing 5 users, raw: true; api users says api users: missing command (one of: list) on standard error and exits 2.

A builder, not a table. A command does not carry its handler; instead every declaration answers a handle, an i32, and the program reads a value by the handle it declared rather than by a name a typo could miss. Each command gets an id you choose, which a switch on r.id dispatches on.

A mistake in the declarations — a flag declared twice in one scope, a bad spec, a required argument after an optional one, a command id used twice or negative, a required flag on a command that holds commands or with a default, a flag a --no- negation would shadow — panics at the call that made it, naming it. A module constant can only be a number, a boolean or a string, so build the tree in main or in a class's constructor.

Declaring. new Cli(name, version, description) is the root, ROOT. group(parent, name, description) adds a command that only holds commands; command(parent, name, id, description) adds one the program runs. alias(handle, name) gives either a second name. flag(cmd, spec, description) declares a flag of one command; globalFlag(spec, description) one every command accepts, before, between or after the verbs.

argument(cmd, spec, description) declares the next positional argument. defaultValue, required, repeatable and negatable refine a flag, and epilog(handle, text) ends a command's help with a paragraph. -h/--help (anywhere before --), -v/--version (at the root) and the word help as the first command are built in; help users kyes, naming no command, is the unknown-command error rather than help.

Spec Means
--all a boolean, given as --all
-a, --all the same, also as -a; short flags cluster (-ar)
--limit <n> takes a value: --limit 5, --limit=5, -l5, -l=5, -l 5; --limit= and -l= give ""
--format [fmt] the value is optional: --format json; given none, ""
--no-color on by default; --no-color turns it off, --color back on; a short (-C, --no-color) turns it off
<id> / [id] a required / optional argument
<file...> / [file...] every argument left; only the last one

Reading the answer. parse(argv, start) reads argv from index start (1 for process.argv, past the program) and never fails: it answers a ParseResult.

Member Answers
kind PARSE_OK, PARSE_HELP, PARSE_VERSION or PARSE_ERROR
id, command, path the reached command's id (-1 for a group), its handle and its command line
flag(f) a boolean's value; for a flag with a value, whether it was given
value(f) the last value given, else the default, else null
values(f) every value of a repeatable flag, in order
source(f) SOURCE_ARGV, SOURCE_DEFAULT or SOURCE_UNSET
positional(a), positionals(a) an argument's value; every value from it on
rest() the arguments after --, which are also positionals
message, exitCode the error or version line; 2 for an error, else 0
hint, errorPath for an error, the pointer to the help and the command it names
help(style) the help of the reached command

source tells a value the user typed from a default, which is what a rule the tree cannot state reads, such as two flags that may not be given together. finish(r, outStyle, errStyle) prints help or the version on standard output and exits 0, prints an error and a pointer to the help on standard error and exits 2, and returns for PARSE_OK. The error line is printed plain; errStyle paints only <path> --help in the pointer:

api users: unknown command 'kyes' (did you mean 'keys'?)
Run 'api users --help' for usage.

The first error wins and names the command it was found at; after a flag error the walk still goes down the command words, so a later -h answers the deepest command, and only an unknown command stops it. A misspelt command or long flag gets the nearest declared one within two edits, or a third of its length if that is more (a swap of two letters is one edit), so --lmt finds --limit; an unknown short flag gets no guess.

Help lists a command's arguments, its own flags and the global ones, descriptions lined up at the widest flag, up to column 32. It appends a default as (default: …), and (required) and (repeatable) to the flags declared so, so a description need not say it.

What it costs and reaches. Declaring builds no help text, and parsing looks each argument up once in a map: on a tree of 500 commands with 8 flags each, a parse takes about a microsecond. Help, guesses and error wording are worked out only when they are printed. parse reaches no capability; finish writes to standard output or standard error and exits. Shell completion and man pages are not generated yet, and nish's own command line does not use this module: src/ can only use what a released compiler ships, so it moves onto nish/cli-args once a release carries it. The example under examples/api-cli/ is a whole client.

nish/release

nish/release tells a command-line tool whether a newer release of it is out on GitHub: the check behind nish upgrade and the compiler's update notice.

import { releaseCheckCached } from "nish/release";

const newer: string | null = releaseCheckCached(
  "https://github.com/owner/tool", "1.2.0",
  "/home/me/.cache/tool/latest", 86400000, 2000);
if (newer !== null) {
  console.error(`tool ${newer} is available (you have 1.2.0)`);
}

releaseLatest(base, timeoutMs) asks <base>/releases/latest once and reads the version from where it redirects, without following it and without a token. It answers null for anything but a redirect to exactly <base>/releases/tag/v<version>: a server that is down, slow, refuses, or sends you to a login page. releaseCheckCached keeps the answer in the file you name, so a second call within maxAgeMs asks nothing; it believes the file only when it is a regular file you own that nobody else can write, and treats one dated in the future as stale.

releaseIsNewer(candidate, current) is true only when both are semantic versions and the candidate is strictly newer, so a bad answer never proposes a downgrade. releaseValidVersion(v) says whether a version may be put in a URL, by install.sh's rule: a digit first, then only letters, digits, ., - and +. releaseFetchInstaller(rawBase, version, path, timeoutMs) writes a release tag's install.sh to a file for you to run with sh; nothing fetched is piped into a shell.

The base must be https://, or http:// to this machine for a test server. The module is native only: it reaches the network and reads file owners. What it defends against is in its security record.

nish/term

nish/term is for a program that draws the whole terminal, many times a second: a game, a dashboard. It does no input or output of its own. A Screen turns what you drew into the string that updates the terminal, and you print it with write. A KeyDecoder turns the bytes you read into key events. The terminal's size and the clock are numbers you pass in. Because the module reaches no capability, a program that uses it builds under --deny of all of them and prints the same bytes under Node. Reading the keys, raw mode and the terminal's size come from nish:term.

To have all of it done for you — opening the terminal, asking it what it can draw, drawing a framebuffer with kitty graphics, Sixel, half blocks or ASCII, and slowing the frame rate when the link cannot keep up — use runTerminal from nish/term-io, which Terminal programs walks through. This section is the layer underneath, for a program that wants its own loop.

import { ATTR_BOLD, COLOR_DEFAULT, Canvas, Screen, hideCursor, rgb, showCursor } from "nish/term";
import { write } from "nish:io";

export const main = (): i32 => {
  const screen = new Screen(20, 4, false);
  const canvas = new Canvas(screen); // 20 x 8 pixels, two to a cell
  canvas.fillRect(2, 2, 6, 2, rgb(0, 160, 255)); // a brick
  canvas.line(0, 7, 19, 7, rgb(90, 90, 90)); // the floor
  screen.text(11, 0, "score 0", rgb(255, 255, 0), COLOR_DEFAULT, ATTR_BOLD);
  write(hideCursor());
  write(screen.flush()); // the first frame: every cell
  screen.text(17, 0, "5", rgb(255, 255, 0), COLOR_DEFAULT, ATTR_BOLD);
  write(screen.flush()); // the next one: a cursor move and the "5"
  write(`${showCursor()}\n`); // a flush ends in the default style
  return 0;
};

A Screen is width × height cells, each a code point, a foreground colour, a background colour and the attributes ATTR_BOLD and ATTR_INVERSE. Draw with set(x, y, codePoint, fg, bg, attrs) and text(x, y, s, fg, bg, attrs), which returns the column after the text, and blank the frame with clear(background). A code point the terminal cannot show in one cell, such as a control character (C0 or C1, so U+009B cannot open an escape) or a surrogate, is sent as U+FFFD, and attribute bits other than those two are dropped. The screen also remembers what it last sent.

flush() answers only the difference: it moves the cursor to the cells that changed and sends a style only where it differs from the cell before. An empty answer means nothing changed. On a 200 × 60 screen, rewriting every cell flushes in about 0.1 ms, and one changed cell in about 0.03 ms. After resize(width, height) the next flush redraws everything, and so does one after invalidate(): call that when something else has written to the terminal. A side is at most 4,096 cells and a screen at most 1,048,576, with rows dropped to fit, so any size a terminal reports is safe to pass.

A Canvas puts pixels on a screen, two to a cell, one above the other. It draws with the half-block character "▀": the foreground is the top pixel's colour and the background is the bottom pixel's. A cell is about twice as tall as it is wide, so the pixels come out roughly square. It has setPixel(x, y, colour), pixel(x, y), fillRect(x, y, w, h, colour), line(x0, y0, x1, y1, colour), clear(colour), and text(column, row, s, fg), which writes over the pixels in cell coordinates. Anything drawn off the edge is dropped.

Colour is rgb(r, g, b), a 24-bit i32, or COLOR_DEFAULT for the terminal's own colour. Some terminals cannot show 24-bit colour; for them, make the screen with new Screen(w, h, true) and every colour is sent as its nearest entry in the 256-colour palette (paletteIndex). Most terminals set COLORTERM to truecolor or 24bit when they can, and reading it is up to your program. The other escapes are plain functions that answer a string: enterAltScreen and leaveAltScreen, hideCursor and showCursor, clearScreen, resetStyle, moveCursor(x, y), and beginSync and endSync, described below.

A KeyDecoder reads the bytes a terminal in raw mode sends. feed(bytes, count, nowMs) answers a KeyEvent[], each with a kind (one of KEY_UP, KEY_DOWN, KEY_LEFT, KEY_RIGHT, KEY_ENTER, KEY_SPACE, KEY_ESCAPE, KEY_CTRL_C, KEY_TAB, KEY_BACKSPACE, KEY_CHAR, KEY_CONTROL, KEY_UNKNOWN) and a code, which for KEY_CHAR is the character's code point; bytes that are not valid UTF-8, and a C1 control (U+0080–U+009F), come out as U+FFFD. A read can stop in the middle of an arrow key's ESC [ A; the decoder holds the start and finishes the key on the next feed.

The ESC key sends a lone ESC, which is also how an arrow key begins. So the decoder holds an ESC until either the rest of a sequence arrives or the timeout you gave the constructor passes. While waiting() is true, call poll(nowMs) once the timeout is up to collect the KEY_ESCAPE. An ESC [ followed by more than 32 parameter bytes is no key, so the decoder gives the ESC up as KEY_ESCAPE at once rather than holding what follows.

import { KEY_LEFT, KeyDecoder } from "nish/term";

export const main = (): i32 => {
  const keys = new KeyDecoder(toF64(30)); // give up on a lone ESC after 30 ms
  const first: u8[] = [toU8(27), toU8(91)]; // ESC [, and the read ends
  const rest: u8[] = [toU8(68)]; // D
  const none = keys.feed(first, 2, toF64(0));
  const left = keys.feed(rest, 1, toF64(4));
  console.log(`${none.length} ${left.length} ${left[0].kind === KEY_LEFT}`); // 0 1 true
  return 0;
};

Events. feed knows keys only. feedEvents(bytes, count, nowMs) and pollEvents(nowMs) read the same bytes as one stream of TermEvents, which is what a program that turns on the mouse, bracketed paste, focus reports or the kitty keyboard protocol needs, and what nish/term-io hands update. An event's type says which fields mean something:

type What it is Its fields
EVENT_KEY a key, legacy, xterm's modified keys, modifyOtherKeys or the kitty keyboard protocol key (a KEY_ kind, including KEY_F1–KEY_F12, KEY_HOME, KEY_END, KEY_PAGE_UP, KEY_PAGE_DOWN, KEY_INSERT, KEY_DELETE), code, modifiers, action, text (kitty's associated text)
EVENT_MOUSE an SGR mouse report (mode 1006, or 1016 in pixels) button (MOUSE_NONE, MOUSE_LEFT, MOUSE_MIDDLE, MOUSE_RIGHT, MOUSE_WHEEL_UP, MOUSE_WHEEL_DOWN, MOUSE_WHEEL_LEFT, MOUSE_WHEEL_RIGHT, MOUSE_BUTTON_8), action, modifiers, the cell x, y and, under 1016, the pixel px, py (else -1)
EVENT_PASTE everything between ESC [ 200 ~ and ESC [ 201 ~, never typed as keys text
EVENT_FOCUS ESC [ I or ESC [ O (mode 1004) gained
EVENT_REPLY the terminal answering a query: DA, DSR, a cursor report, DECRPM, XTVERSION, DECRQSS, OSC, APC. Never a key introducer ([, P, _, ], ^ or X), prefix (?, >, =, < or 0), params, intermediate, the final byte reply, text
EVENT_RESIZE, EVENT_QUIT never decoded: the code that watches the size makes the first, KeyState the second —

modifiers is a sum of MOD_SHIFT (1), MOD_ALT (2), MOD_CTRL (4), MOD_SUPER (8), MOD_HYPER (16) and MOD_META (32); action is ACTION_PRESS, ACTION_REPEAT, ACTION_RELEASE or, for the mouse, ACTION_MOVE. ESC [ row ; col R is both a cursor position report and F3 with modifiers, so a program that sends ESC [ 6 n adds 1 to the decoder's cprOutstanding and the next R comes back as the reply.

Only the kitty keyboard protocol reports a key coming up. A KeyState says which keys are held whatever the terminal sends: apply(events, nowMs) answers the events with a release made up for a key that has not repeated for releaseGapMs (100 by default) where the terminal sends none, a held key's repeats marked ACTION_REPEAT, and Ctrl-C turned into EVENT_QUIT while quitOnCtrlC is set; tick(nowMs) answers the releases that fall due between reads, and isDown(key, code) whether a key is held now.

Made-up releases are only as good as the bytes arriving on time, so delivered(sentAtMs) tells it that the answer to a query sent at sentAtMs has come back: a key is then let go only once an answer sent releaseGapMs after its last repeat is in, since a key still held would have repeated ahead of it.

import { EVENT_KEY, EVENT_REPLY, KeyDecoder } from "nish/term";

export const main = (): i32 => {
  const keys = new KeyDecoder(toF64(30));
  // `a`, then a DA1 answer: ESC [ ? 6 2 c
  const bytes: u8[] = [toU8(97), toU8(27), toU8(91), toU8(63), toU8(54), toU8(50), toU8(99)];
  const events = keys.feedEvents(bytes, toI32(bytes.length), toF64(0));
  const key = events[0];
  const reply = events[1];
  console.log(`${key.type === EVENT_KEY} ${key.code} ${reply.type === EVENT_REPLY} ${reply.params[0]}`); // true 97 true 62
  return 0;
};

At 120 or 240 Hz, two things matter. The first is that the terminal never shows a frame half drawn. Set screen.sync = true and every flush that changes something is wrapped in beginSync() and endSync() (synchronized output, mode 2026). A terminal that supports the mode holds the frame and paints it in one go; kitty, foot, WezTerm and iTerm2 do, and one that does not ignores the escapes. An unchanged frame stays an empty string. The second is that the game runs at the same speed whatever the display does.

A FixedStep clock turns the time between frames into whole steps: new FixedStep(stepsPerSecond, maxSteps), then advance(nowMs) answers how many steps to run and keeps the remainder for next time. alpha() is that remainder as a fraction of a step, for drawing between the last two positions. A clock that falls more than maxSteps behind, after a stall, drops the backlog instead of trying to catch up. The same clock with maxSteps 1 paces the drawing: it answers 1 when a frame is due. The first advance only starts the clock, and answers 0.

import { COLOR_DEFAULT, Canvas, FixedStep, Screen, rgb } from "nish/term";
import { write } from "nish:io";
import { monotonicNanos } from "nish:process";

export const main = (): i32 => {
  const screen = new Screen(40, 12, false);
  screen.sync = true; // each frame is painted whole
  const canvas = new Canvas(screen); // 40 x 24 pixels
  const logic = new FixedStep(240, 8); // the game moves 240 times a second
  const draw = new FixedStep(120, 1); // and is drawn 120 times a second
  const start = monotonicNanos();
  let x = 0;
  let nowMs = toF64(0);
  while (nowMs < toF64(2000)) {
    nowMs = toF64(monotonicNanos() - start) / toF64(1000000);
    const steps = logic.advance(nowMs);
    for (let k = 0; k < steps; k++) {
      x = (x + 1) % 480; // one pixel every 6 steps
    }
    if (draw.advance(nowMs) > 0) {
      canvas.clear(COLOR_DEFAULT);
      canvas.fillRect((x / 6) | 0, 10, 4, 4, rgb(255, 120, 0));
      write(screen.flush());
    }
  }
  return 0;
};

The loop above spins between frames to stay short; a real one waits on its keyboard input until the next frame is due. A frame of a game like this flushes in well under a millisecond, so the budget at 240 Hz, about 4 ms, goes to the terminal, which has its own pacing. kitty waits repaint_delay milliseconds (10 by default) before it paints; repaint_delay 2 and sync_to_monitor yes in kitty.conf let it keep up. foot paints on the compositor's frame callback, so it follows the monitor's refresh rate by itself.

nish/term-pixels

nish/term-pixels is the part of nish/term-io that turns pixels into terminal output. It is pure, with no capability and no I/O, so a program that wants to write its own loop, or test what it draws, can use it alone.

  • Framebuffer(width, height) is a grid of 0xRRGGBB pixels, black when made, with set, get, fill, clear and resize. A pixel set off the edge is not drawn. scaleFramebuffer(src, divisor, dst) shrinks one by a whole divisor.
  • GRAPHICS_ASCII, GRAPHICS_HALFBLOCK, GRAPHICS_SIXEL and GRAPHICS_KITTY (0 to 3) name the backends, from worst to best, so a terminal's backend and a cap combine with Math.min.
  • fitFramebuffer(fbW, fbH, cols, rows, cellW, cellH, backend) answers the PixelRect to draw in: the cells the picture covers, centred in a cols × rows area with the aspect ratio kept, and its size in the backend's pixels. A cell size of 0 means "unknown" and takes a usual one.
  • The four encoders draw a framebuffer into a rect, sampling it nearest-neighbour; none of them clears the letterbox around it:
    • kittyFrame(fb, rect, imageId, out) writes kitty graphics commands into a PixelBytes. Each frame resends the same image id, which the terminal swaps in whole; kittyDelete(imageId) frees it at the end.
    • sixelFrame(fb, rect, dither, out) writes one Sixel image in a fixed palette of SIXEL_REGISTERS colours, ordered-dithered when dither is true.
    • drawHalfBlocks(fb, canvas, rect) draws onto a Canvas, two pixels to a cell.
    • drawAscii(fb, screen, rect, colors) draws onto a Screen, one luminance character to a cell.

A PixelBytes keeps the kitty and Sixel encoders' output and working arrays, grown once to the largest frame, so a program that keeps one encodes frame after frame without allocating. out.text() is the bytes to write and out.reset() empties it for the next frame.

import { GRAPHICS_KITTY, Framebuffer, PixelBytes, fitFramebuffer, kittyFrame } from "nish/term-pixels";

export const main = (): i32 => {
  const fb = new Framebuffer(64, 32);
  fb.fill(0x203040);
  fb.set(10, 5, 0xffcc00);
  // A 64×32 picture in 40×20 cells of 10×20 pixels.
  const rect = fitFramebuffer(fb.width, fb.height, 40, 20, 10, 20, GRAPHICS_KITTY);
  console.log(`cells ${rect.col},${rect.row} ${rect.cols}x${rect.rows}, ${rect.width}x${rect.height} px`);
  // cells 0,5 40x10, 400x200 px: letterboxed top and bottom
  const out = new PixelBytes(0);
  kittyFrame(fb, rect, 1, out);
  console.log(`${out.length > 0}`); // true: out.text() is ready to write
  return 0;
};

nish/input

nish/input names what one fixed step of a program reads from its player: the TermEvents since the last step and the KeyState of which keys are held. It is pure, with no capability, no I/O and no clock, so a step that takes a StepInput compiles the same for a terminal, for wasm32 behind a page, or for a test that scripts the keys. The events and key state are nish/term's.

A host owns one StepInput and points its events and keys at what it read each step; nothing is copied. A terminal program points them at the TermInput that nish/term-io's runTerminal hands it.

import { KEY_CHAR, KeyState } from "nish/term";
import { StepInput } from "nish/input";

const quits = (input: StepInput): boolean => input.isDown(KEY_CHAR, 113);

export const main = (): i32 => {
  const input = new StepInput(new KeyState());
  console.log(quits(input));
  return 0;
};

nish/raster

nish/raster draws into a frame of 0xRRGGBB pixels: an Int32Array read row by row, width pixels a row, height rows. Every draw takes the frame and its size first, clips to the frame, and never writes past the rows the array holds, so a frame too short for its size is safe. It is pure, with no capability, no I/O and no clock.

The same draw fills either kind of frame, with no copy:

  • a terminal program's Framebuffer from nish/term, whose pixels array rasterFrame lends as the Int32Array a draw takes;
  • module storage, const FRAME = new Int32Array(...), which is how a wasm32 module shares a frame with its page (nish/host/game).
import { RASTER_SOLID, rasterDisc, rasterFrame, rasterGradient, rasterRect, rasterText } from "nish/raster";

export const main = (): i32 => {
  const width: i32 = 64;
  const height: i32 = 32;
  const pixels: i32[] = new Array<i32>(width * height);
  pixels.fill(0);
  const frame = rasterFrame(pixels);
  rasterGradient(frame, width, height, 0x101830, 0x302040);
  rasterRect(frame, width, height, 4, 20, 28, 24, 0xffffff, RASTER_SOLID);
  const r: f64 = 6.0;
  rasterDisc(frame, width, height, 48.0, 12.0, r, 0xff8000, 1.0);
  rasterText(frame, width, height, 2, 2, 1, "HI", 0xffff00);
  console.log(pixels[22 * width + 10] === 0xffffff);
  return 0;
};

Colours are 0xRRGGBB integers: rasterRgb packs three channels, rasterMix moves one colour toward another, rasterAdd adds light, rasterDim darkens and rasterHue walks the rainbow. A weight of 0 to RASTER_SOLID (256) says how much of a colour a rectangle blends in. RasterTrail and RasterParticles are made once with a capacity and drawn every frame, so a frame allocates nothing. Text is the 5×7 font, one glyph a byte, and a byte outside printable ASCII draws as ?.

nish/synth

nish/synth makes short sounds out of arithmetic. synthSilence makes an array of zero samples, and each synthTone adds one voice into it: a square, triangle or noise wave whose pitch slides from f0 to f1 Hz and whose loudness falls away by decay a second. The sample rate is passed to every call, so one program can make sounds at whatever rate its player wants.

import { SYNTH_SQUARE, SYNTH_TRIANGLE, synthSemitone, synthSilence, synthTone } from "nish/synth";

export const main = (): i32 => {
  const rate: i32 = 22050;
  const out: f64[] = synthSilence(0.2, rate);
  synthTone(out, rate, 0.0, 0.2, 440.0, 220.0, SYNTH_SQUARE, 0.25, 8.0);
  synthTone(out, rate, 0.05, 0.1, synthSemitone(440.0, 7), synthSemitone(440.0, 7), SYNTH_TRIANGLE, 0.2, 12.0);
  console.log(out.length);
  return 0;
};

The samples are not clipped: a voice stays within its volume, so voices whose volumes add up to at most 1 stay inside -1 to 1. Noise is a fixed sequence, so the same calls give the same samples on every run.

nish/host/game

nish/host/game is the memory a game built with --target wasm32 shares with the page that runs it. Each piece is module storage, which the page reads and writes in place through the loader's views().storage:

Storage Written by Holds
GAME_FRAME the game up to 960×540 0xRRGGBB pixels, width a row
GAME_INFO the game the frame's width and height, then the last sound's sample count and rate
GAME_INPUT the page a nish/host/ring of 7-word key, pointer and focus records
GAME_SOUNDS the game a ring of sound ids to start
GAME_SAMPLES the game one sound's samples, as f32

Each step, the game calls gameReadInput to drain the page's records into a nish/input StepInput, so it reads keys with the same nish/term constants a terminal program uses. It draws into GAME_FRAME with nish/raster and calls gamePlaySound for each sound to start. At start, the page calls an export of the game's once for each sound id, which makes the sound and writes it with gameWriteSamples. The page then copies the samples into an audio buffer. Each record's words, and the page's side of every piece, are listed in the module's comment and in the library reference.

The input differs from a terminal's in two ways. Ctrl+C is an ordinary key (on a page it is copy), never EVENT_QUIT. The reader makes up no auto-repeat either: a page that wants held keys to repeat sends ACTION_REPEAT records itself.

To run on a page, compile with --target wasm32 and link with scripts/build.sh --profile wasm-host (In a browser), which exports the storage for the loader. Pin the memory with --memory <MiB> so the page's views of it never go stale: the storage alone is about 2.5 MB (the frame is 2 MB), so give it at least 4 MiB plus what the game allocates. The GAME_* names are the keys the page reads, so keep them for this module. When two modules declare storage of one name, the later one's is keyed by its symbol instead, and a program's own GAME_FRAME could move this module's off its key.

import { GAME_FRAME, gameHeight, gamePlaySound, gameReadInput, gameSetSize, gameWidth } from "nish/host/game";
import { RASTER_SOLID, rasterRect } from "nish/raster";
import { StepInput } from "nish/input";
import { KEY_CHAR, KeyState } from "nish/term";

const step = (input: StepInput, nowMs: f64): void => {
  gameReadInput(input, nowMs);
  if (input.isDown(KEY_CHAR, 32)) {
    gamePlaySound(0);
  }
  rasterRect(GAME_FRAME, gameWidth(), gameHeight(), 0, 0, gameWidth(), gameHeight(), 0x101830, RASTER_SOLID);
};

export const main = (): i32 => {
  const input = new StepInput(new KeyState());
  gameSetSize(320, 180);
  step(input, 0.0);
  console.log(GAME_FRAME[0] === 0x101830);
  return 0;
};

nish/base64

nish/base64 writes standard base64 (RFC 4648 §4): the alphabet A-Z a-z 0-9 + /, four characters for every three bytes and a short last group filled out with =. It is what kitty's graphics protocol carries an image in.

base64Encode(bytes) answers a string; base64EncodeInto(src, off, len, dst, at) writes the encoding of src[off..off+len) into dst from at, allocating nothing, and answers the index after the last character, for a program that builds its output in one buffer. base64EncodedLength(n) is how many characters n bytes take, so the buffer can be sized first. A range outside its array panics before anything is written. The module only encodes; the URL-safe, unpadded decoder for tokens and keys is nish/crypto/base64url.

import { base64Encode, base64EncodeInto, base64EncodedLength } from "nish/base64";

export const main = (): i32 => {
  const bytes: u8[] = [toU8(110), toU8(105), toU8(115), toU8(104)]; // "nish"
  console.log(base64Encode(bytes)); // bmlzaA==
  const out: u8[] = new Array<u8>(base64EncodedLength(3));
  const end = base64EncodeInto(bytes, 0, 3, out, 0);
  console.log(`${end} ${out[3]}`); // 4 122, the "z" of "bmlz"
  return 0;
};

nish/bytes

A file header, a network packet or a hash's input is a u8[] with numbers laid out in it. nish/bytes reads one from, or writes one into, the array at an offset, in either byte order: LE (little-endian) puts the least significant byte at the offset, BE (big-endian, network order) the most significant.

Width Read Write
u16 getU16LE, getU16BE setU16LE, setU16BE
u32 getU32LE, getU32BE setU32LE, setU32BE
u64 getU64LE, getU64BE setU64LE, setU64BE
i32 getI32LE, getI32BE setI32LE, setI32BE
i64 getI64LE, getI64BE setI64LE, setI64BE
f64 getF64LE, getF64BE setF64LE, setF64BE

A signed width is the same bytes read as two's complement, and an f64 is its IEEE 754 bits. There is no f32.

import { getU16LE, getU32BE, setU32LE } from "nish/bytes";

export const main = (): i32 => {
  const b: u8[] = [0, 0, 1, 0, 0, 0, 0, 0];
  console.log(`${getU32BE(b, 0)}`); // 256
  setU32LE(b, 4, toU32(513));
  console.log(`${b[4]} ${b[5]}`); // 1 2
  console.log(`${getU16LE(b, 4)}`); // 513
  return 0;
};

Every access is an ordinary index. An offset whose bytes do not all lie in the array panics with index out of range, naming the first byte that is not there. A guard in your own loop, such as off + 3 < b.length, does not yet remove these checks, so a hot loop that needs them gone indexes the bytes itself.

Natively every function answers for every value. Under Node the u16 and u32 functions are expected to answer the same, but the 64-bit ones, the i32 ones and f64 do not yet: the Node prelude's conversions saturate where native ones wrap.

Runs and copies

A parser spends its time skipping: the blanks between tokens, the digits of a number, the plain text inside a string; a writer skips the bytes it can copy without an escape. Four functions answer where such a run ends — the first index in [from, to) whose byte is not in the class, or to if every byte is:

Function Skips
skipBlank(b, from, to) JSON's blanks: space, tab, newline, carriage return
skipDigits(b, from, to) the ASCII digits 0 to 9
skipJsonText(b, from, to) what a JSON string holds as itself: printable ASCII other than " and \, and well-formed UTF-8
skipJsonPlain(b, from, to) what a JSON writer copies as it is: every byte from 0x20 up other than " and \, with no UTF-8 check

copyBytes(dst, at, src, from, count) copies count bytes of src from from into dst at at. The two arrays may be the same one, and the ranges may overlap.

import { copyBytes, skipBlank, skipDigits } from "nish/bytes";

export const main = (): i32 => {
  const text: u8[] = [32, 10, 52, 50, 44]; // " \n42,"
  const start = skipBlank(text, 0, 5);
  const end = skipDigits(text, start, 5);
  console.log(`${start} ${end}`); // 2 4
  const digits: u8[] = [0, 0];
  copyBytes(digits, 0, text, start, end - start);
  console.log(`${digits[0]} ${digits[1]}`); // 52 50
  return 0;
};

The runs never panic: to is clamped to the array and from to to, so a range past the end answers the end. copyBytes panics with copyBytes: the range lies outside an array if either window does not fit, before it writes anything. skipJsonText stops at a quote, a backslash, a control byte or the first byte of a malformed UTF-8 sequence — the byte your decoder has to look at next. skipJsonPlain stops at a quote, a backslash or a control byte — the byte your writer has to escape.

Natively each run reads 16 or 32 bytes at a time and the copy is one block move; under Node they are plain loops with the same answers. The runs take longer the longer the run is, so they are for text, never for secret data.

nish/sqlite

A SQLite database in a file or in memory: openDatabase for a Database, exec and prepare for a Statement you bind, step and reset as often as you like, and every failure a Result. SQLite itself is compiled into your program, so there is nothing to install. It has a page of its own, SQLite.

nish/threads

Running work on several threads: parallelMapInto, parallelMap, parallelFrom and parallelReduce for one function over an array, scope() for a few different tasks at once, and Mutex<T> and Channel<T> for sharing between them. Importing the module is all it takes to build a threaded program. It has a page of its own, Threads.

nish/sync/shm and nish/sync/ring

Memory several processes share: createRegion and openRegion map a named region, read and written through checked atomic slots (loadI32, addI64, compareExchangeI32, ...) and byte windows, and waitI32 and wake sleep and wake across processes on Linux. nish/sync/ring builds bounded queues of byte records in a region: SpscRing for one producer and one consumer, MpscRing for many producers. Native only. It has a page of its own, Shared memory between processes. Records can also be rendered and sent in place (reserve, commit, peek, Http1Connection.sendBody, release) rather than copied in and out.

nish:secret

nish:secret is written with a colon because it is a builtin module, but it ships beside the library and the cryptography uses it. A Secret<T> holds key material — an array of integers, or a record of integer fields — and the compiler refuses every road out of it but one: expose(s, f) or exposeWith(s, arg, f) hands the value to a function. wipe(s) zeroes it with a store the optimiser may not remove.

import { Secret, exposeWith, secret, wipe } from "nish:secret";
import { hmacSha256 } from "nish/crypto/hmac";

const tag = (key: u8[], message: u8[]): u8[] => hmacSha256(key, message);

export const main = (): i32 => {
  const keyBytes: u8[] = [1, 2, 3, 4];
  const key: Secret<u8[]> = secret(keyBytes); // `keyBytes` is moved in
  const message: u8[] = [104, 105];
  const mac: u8[] = exposeWith(key, message, tag);
  wipe(key);
  console.log(`${mac.length}-byte tag`); // 32-byte tag
  return 0;
};

Printing a Secret, putting it in a template, or handing it to any builtin is a compile error.

The type can be left off: secret answers a Secret of whatever it is given, and every rule below holds the same.

import { expose, secret, wipe } from "nish:secret";

const length = (k: u8[]): i32 => toI32(k.length);

export const main = (): i32 => {
  const key = secret(new Array<u8>(32)); // a `Secret<u8[]>`
  console.log(`${expose(key, length)} bytes`); // 32 bytes
  wipe(key);
  return 0;
};

secret(v) takes a value nothing else holds: a literal, a call that makes a new array or record, or a local that is the only name for its value. A second name made before the move would read the key outside the Secret, and zeros once it is wiped, so the compiler refuses it and says where it was made:

import { Secret, secret, wipe } from "nish:secret";

export const main = (): i32 => {
  const raw: u8[] = [1, 2, 3];
  const alias = raw; // a second name for the same bytes
  const key: Secret<u8[]> = secret(raw);
  wipe(key);
  return toI32(alias[0]);
};

Storing the array in a field or an element, pushing it, or handing it to a function that keeps it is a second name too. To keep the plain bytes as well, copy them first, and move the copy in:

import { Secret, secret, wipe } from "nish:secret";

export const main = (): i32 => {
  const raw: u8[] = [1, 2, 3];
  const copy: u8[] = new Array<u8>(raw.length);
  copy.set(raw);
  const key: Secret<u8[]> = secret(copy); // `copy` is the only name for its bytes
  wipe(key);
  console.log(`${raw[0]}`); // 1: `raw` is its own array
  return 0;
};

The rules are in Secrets.

Keeping a key in an object

A key that lives as long as a connection or a session can be a field. The field is readonly and exactly Secret<T>, the class is not generic, and the class has a wipe(): void that calls wipe(this.field) for every such field on every path: the checker reads that method, and a field it can leave unwiped is a compile error.

import { Secret, expose, secret, wipe } from "nish:secret";

const count = (k: u8[]): i32 => toI32(k.length);

class Session {
  readonly key: Secret<u8[]>;
  constructor() {
    this.key = secret(new Array<u8>(32)); // set once, from `secret(...)`
  }
  size(): i32 {
    return expose(this.key, count); // lent to `expose`, not copied out
  }
  wipe(): void {
    wipe(this.key);
  }
}

export const main = (): i32 => {
  const s = new Session();
  console.log(`${s.size()}-byte key`); // 32-byte key
  s.wipe();
  return 0;
};

The field is set only in the constructor, only from secret(...), and only once. It may be read only to lend it: as an argument of a call or of new, or as an operand of === or !==. Copying it into a local, returning it, or storing it anywhere else is refused:

import { Secret, secret, wipe } from "nish:secret";

class Session {
  readonly key: Secret<u8[]>;
  constructor() {
    this.key = secret(new Array<u8>(32));
  }
  wipe(): void {
    wipe(this.key);
  }
}

export const main = (): i32 => {
  const s = new Session();
  const k = s.key; // a second name for the key, outside the holder
  s.wipe();
  return 0;
};

The holder's wipe() is the only place the field ends: wipe(s.key) anywhere else is refused, so call s.wipe() instead. A function that takes a Secret borrows it, so it may lend it on or expose it, but it neither returns it nor wipes it:

import { Secret, secret, wipe } from "nish:secret";

const same = (k: Secret<u8[]>): Secret<u8[]> => k; // a second name for the caller's key

export const main = (): i32 => {
  const k = secret(new Array<u8>(32));
  const again = same(k);
  wipe(again);
  return 0;
};

Up to 0.18.0 a function could return or wipe a Secret it was passed; from the release after it, both are refused. Drop such a return and keep using the key you passed in, and leave the wipe to the code that made the key.

Calling wipe() is the program's job: the compiler checks that the method wipes every field, not that something calls it. The rules are in A Secret field.

nish/crypto

One module per primitive, each imported by its own name, each written in Nish from its specification and checked against that specification's published test vectors. A byte string is a u8[].

Module Main exports
nish/crypto/sha256 sha256(data), and Sha256, a streaming hasher (update, copy, digest, and digestInto(out, at), which writes the digest into your array)
nish/crypto/sha512 sha512, sha384, and the streaming Sha512 and Sha384
nish/crypto/hmac hmacSha256, hmacSha384, hmacSha512, their streaming classes (HmacSha256.digestInto(out, at) writes the tag into your array and allocates nothing), and hmacSha256Verify and its siblings, which compare in constant time; and pbkdf2HmacSha256(password, salt, iterations, dkLen), PBKDF2 (RFC 8018), which takes the password as a Secret<u8[]>, answers the key as one, and answers null for fewer than one iteration or a dkLen below one
nish/crypto/hkdf hkdfExtractSha256, hkdfExpandSha256 and their SHA-384 forms; TLS 1.3's hkdfExpandLabelSha256; and a Secret form of each (hkdfExtractSha256Secret, hkdfExpandLabelSha256Secret, …), which takes the key as a Secret<u8[]>, answers one, and wipes what it derives on the way; and hkdfExpandLabelSha256Held(dst, prk, label, context) and its SHA-384 form, which derive into a Secret you already hold, as many bytes as it holds, in place — dst may be prk, a ratchet
nish/crypto/ct timingSafeEqual(a, b), a comparison that reads every byte
nish/crypto/bcrypt bcryptHash(password, salt, cost) and bcryptVerify(password, hash), bcryptHashN/bcryptVerifyN for up to six at once at a width bcryptCalibrate picks, and bcryptHash4/bcryptVerify4 for up to four: bcrypt password hashes that other implementations read, with the password a Secret<u8[]>; see Password hashing
nish/crypto/bcrypt-batch bcryptBatcher(config): a login batcher that queues verify and hash requests, sends them at the calibrated width or after a hold, and runs the batches on nish/threads tasks across cores; see A login batcher
nish/crypto/base64url base64urlEncode and a strict base64urlDecode, unpadded; base64StdPadEncode and a strict base64StdPadDecode, padded, in the standard alphabet, constant-time (nish/base64 encodes the same alphabet for pixels, without the constant-time guarantee)
nish/crypto/chacha20poly1305 chacha20Poly1305Seal and chacha20Poly1305Open
nish/crypto/aes aesKey for AES-128 and AES-256, aesGcmSeal and aesGcmOpen
nish/crypto/x25519 X25519 key exchange
nish/crypto/ed25519 Ed25519 signatures (RFC 8032): ed25519PublicKey(seed), ed25519Sign(seed, msg) and ed25519Verify(pub, msg, sig); the seed is a 32-byte Secret<u8[]>, and a message longer than 2^31 − 1 bytes panics; verification does not refuse a public key of small order
nish/crypto/p256 P-256 ECDSA signing and verification
nish/crypto/p384 P-384 ECDSA verification
nish/crypto/rsa RSA signature verification, PKCS#1 v1.5 and PSS over SHA-256/384/512, under a PKCS#1 public key: rsaParsePublicKey, rsaVerifyPkcs1v15, rsaVerifyPss
nish/crypto/x509 DER and PEM, X.509 certificates read with their signature algorithm, key type and v3 extensions, and self-signed P-256 certificates
nish/crypto/x509-random a random P-256 key and certificate serial for x509; native only, so a program that imports it does not build for WebAssembly
nish/crypto/pki-roots pkiLoadSystemRoots(), the root certificates to trust, parsed: those SSL_CERT_FILE and SSL_CERT_DIR name when either is set, with no fallback, and otherwise the first system bundle (pkiSystemBundles()) and /etc/ssl/certs. A block or file it cannot use is counted in skipped, never fatal, and only regular files are read; on macOS it reads /etc/ssl/cert.pem, not the Keychain, so roots and trust settings added there are missed. Native only
nish/crypto/pki pkiVerifyServer(chain, hostname, nowMs, store), whether a TLS server's chain leads to a root of pkiTrustStore(roots) and names the host, as a PkiResult of a code (PKI_OK or a PKI_ERR_ reason, pkiReason(code) its sentence) and the path. Checks signatures, validity, CA and path-length rules, key usages, name constraints and the hostname, never the common name; revocation is not checked. Verifying TLS certificates
nish/crypto/sha1 sha1(data), not for security: it is here only because the WebSocket handshake is defined on it
import { base64urlEncode } from "nish/crypto/base64url";
import { sha256 } from "nish/crypto/sha256";

const bytesOf = (text: string): u8[] => {
  const out: u8[] = [];
  const n: i32 = toI32(text.length);
  for (let i: i32 = 0; i < n; i++) {
    out.push(toU8(text.charCodeAt(i)));
  }
  return out;
};

export const main = (): i32 => {
  console.log(base64urlEncode(sha256(bytesOf("abc"))));
  return 0;
};
$ nish run digest.ts
ungWv48Bz-pBQUDeXa4iI7ADYaOWF3qctBD_YfIAFa0

A native program uses the CPU's own instructions for three of these where the CPU has them, and gives the same bytes either way: nish/crypto/sha256 compresses each block with SHA-NI on x86-64 or the ARMv8 SHA-256 instructions on AArch64, nish/crypto/aes runs AES-NI or the ARMv8 AES instructions, and nish/crypto/chacha20poly1305 makes ChaCha20's keystream eight blocks at a time with AVX2 on x86-64, or four with NEON on AArch64 (a message of 256 bytes or more; a shorter one stays on the Nish code).

The program checks the CPU once, when it first hashes or encrypts. Setting NISH_SHA256=soft, NISH_AES=soft or NISH_CHACHA20=soft in the environment the program starts in keeps that primitive on its portable Nish code, which is how to compare the two paths or rule one out; any other value is ignored. A WebAssembly build always takes the portable code.

A password-derived key is a Secret on both sides of pbkdf2HmacSha256, and the program wipes both:

import { pbkdf2HmacSha256 } from "nish/crypto/hmac";
import { Secret, secret, wipe } from "nish:secret";

export const main = (): i32 => {
  const passwordBytes: u8[] = [112, 97, 115, 115, 119, 100]; // "passwd"
  const password: Secret<u8[]> = secret(passwordBytes);
  const salt: u8[] = [115, 97, 108, 116]; // "salt"
  const key: Secret<u8[]> | null = pbkdf2HmacSha256(password, salt, 4096, 32);
  wipe(password);
  if (key === null) {
    return 1;
  }
  console.log("derived 32 bytes");
  wipe(key);
  return 0;
};

A function that returns u8[] | null answers null for an input it refuses — a bad key length, a tag that does not verify — so narrow the result before you use it. The modules that hold a private key (p256, x25519, ed25519, x509, x509-random) take and give it as a Secret<u8[]>. Every module but sha1 and ed25519, which came later, has been through a security audit (ed25519 is recorded as a change after its area's audit, and pki-roots and pki have a record written with their code); the records are in docs/security/.

nish/net

The protocol stack written on nish/crypto: TLS 1.3 (nish/net/tls), QUIC (nish/net/quic and its parts), HPACK and QPACK, HTTP/1.1 (nish/net/http1-codec, nish/net/http1-server), HTTP/2 (nish/net/http2), HTTP/3 (nish/net/http3), WebSocket (nish/net/websocket, the server's and the client's side of its handshake) and WebTransport. Most of these modules are sans-IO: they take bytes and answer bytes, and never read a socket or a clock themselves, so you drive them from your own loop over nish:net. The ones that sit on sockets are nish/net/tls-tcp, nish/net/http1-server (its Http1Server and Http1TlsServer pools), nish/net/http2-tls and nish/net/http3-server; like nish:net itself, they are native only.

Your loop re-arms an Http1Server slot only when server.rearm(slot, wants) says its interest changed, so a keep-alive request costs one recv and one send and no pollModify. A slot borrows its output buffer only while it writes, so an idle connection costs a few KiB, not outputSize. Over plain TCP, Http1Connection.sendBody(buf, off, len) sends a Content-Length body straight from your array, in one sendmsg with whatever output is queued before it, and copies only what the socket did not take; it answers as write does and keeps no hold on buf, so the array is yours again once it returns.

buf is a readonly u8[], as write's is, so a view you may only read, such as a ring's peek, is handed over as it is. A chunked or compressed response, an Http1TlsServer slot, and a write still held back are copied as write copies them.

A response with a Content-Length is finished once write or sendBody has taken its last byte, and one with no body (Content-Length: 0, a HEAD, a 204 or a 304) once respond has queued its head: the next request on a keep-alive connection is answered whether or not you call conn.end(). A chunked body needs end(), which writes its last chunk, and so does a chunked response to an HTTP/1.0 client, which is sent unframed: without it the connection waits and the next request is never answered. Calling end() on any response stays correct; it answers 0 once the length is met and H1_INVALID while bytes are still owed.

A slot lends its input and head buffers back too, whenever it has answered a request and holds no byte of the next: a server holding tens of thousands of keep-alive connections that each send a request now and then keeps those buffers only for the connections mid-request, for two pool operations a request, and an idle connection holds about 1.2 KB. That is Http1Config.idleInput, on by default; set it to false and each slot keeps its own buffers while it rests, about 1.9 KB more a connection. An Http1Server builds a slot's connection on the first accept into it and keeps it for the next, so a pool sized for 16,384 connections costs some 16 KiB until they come; bench/http.mjs load-tests that loop on one core.

Call expire(now) on an Http1Server or Http1TlsServer from your loop, with monotonicNanos(), now and then: it closes a slot that has moved no byte for idleTimeout milliseconds, and one whose request head has been arriving for longer than headTimeout (30 s by default), so a client that trickles its head a byte at a time cannot hold a slot. Over TLS the head's clock starts with the first bytes after the handshake, so a client that finishes its handshake and then sends nothing is closed after headTimeout too. Both are Http1Config fields, and a value of 0 or less stops start-up.

A request Http1Server refuses (a 400, a 413) is not closed at once: the server shuts its write side and reads and throws away what the client still sends, so the client's TCP stack does not answer those bytes with a reset that destroys the error response (RFC 9112 §9.6). The drain stops when the client hangs up, after lingerBytes (64 KiB by default; 0 closes at once), or when expire finds it past lingerTimeout (2 s by default) — only expire reads that deadline, so call it from your loop. Until the linger ends the slot is busy and counts against the pool.

Http2TlsServer has an expire(now) too. A client that has not acknowledged the server's SETTINGS within Http2Config.settingsTimeout (10 s by default, its TLS handshake included) is sent GOAWAY SETTINGS_TIMEOUT and closed; one that moves no byte for Http2Config.idleTimeout (60 s) is sent GOAWAY NO_ERROR and closed. Any frame counts as a byte moved, a PING included, and a quiet stream counts for nothing: a long poll or server-sent event stream with nothing to send for idleTimeout is cut, so send on it (a comment line, for SSE) more often than that, or raise it. A WebTransport session has its own clock: expire(now) ends one quiet for WebTransportConfig.idleTimeout (60 s) — refused with 503 if it was never answered, closed with code 0 if it was — and next reports it as WT_CLOSED. A session takes at most maxDatagramsPerSecond datagrams a second (1,024 by default), counted in fixed one-second windows; the rest are dropped and counted in datagramsDropped.

Calling expire on every turn of the loop is cheap: it walks the slots only when one can be due, about once per the shortest of its timeouts, and expireScans counts the walks. TlsTcpServer, Http1TlsServer, Http2TlsServer and Http3Server build a slot the same way, on the first connection accepted into it (an Http2TlsServer builds slot 0 in its constructor), so a pool's size is a ceiling on its connections. What start-up still takes grows with it: the tables sized from the cap, such as an Http3Server's connection-ID index, timer wheel and rings.

nish/net/tls/client and nish/net/tls/record-client are the client side of TLS 1.3, sans-IO like the server. nish/net/tls/verify checks the server for them (tlsClientAuthenticate), and nish/net/tls-tcp's TlsTcpClient and blocking tlsConnect put the client on a TCP socket and check the server themselves, against the system's roots or ones you name. TLS client shows them.

nish/net/websocket is both sides of a WebSocket's opening handshake: a server accepts with Http1Connection.acceptWebSocket, and a client makes its key with websocketClientKey from 16 random bytes, writes the request with websocketUpgradeRequest, and checks the server's 101 with websocketCheckResponse, which answers a WS_HS_* code and the chosen subprotocol. WebSocket shows both.

nish/net/router maps a request's method and path to a route you added, on any of the three HTTP versions, without allocating; HTTP routing shows it.

nish/net/handler is a request and response layer over Http1Server: an HttpHandler accepts, reads and routes, and your loop takes whole requests (wait, ready, take) and answers each with send, sendJson, problem or a streamed start, write, end, now or later. It owns the socket loop, so it is native only; API server shows it.

nish/net/http-compress reads Accept-Encoding and gzips a response body, and Http1Connection.respondEncoded sends one with Content-Encoding and Content-Length set; Compression shows it.

nish/net/http1-client-codec is the client side of HTTP/1.1: http1RequestHead writes a request's head and Http1ResponseParser reads the response, fed any slice, refusing every response-smuggling shape RFC 9112 names; HTTP client shows it.

nish/net/fetch is a blocking client for command-line programs: fetchSync(request, options) answers the response with a reader for its body, and a FetchSyncClient keeps a connection alive between fetches. Every wait has a timeout, every body a cap, and redirects and proxies are followed only as HttpOptions says; HTTP client shows it.

nish/net/postgres is a PostgreSQL client that leaves the socket to your loop: prepared statements, pipelined groups, SCRAM-SHA-256, and rows read in place from the receive buffer; PostgreSQL shows it.

nish/net/prefork runs a server on every core: preforkWorkers(process.argv) starts the program once per worker, each on its own SO_REUSEPORT listener, restarts the ones that exit and stops them all on SIGTERM. It is native only; Networking shows it.

Every final response an Http1Connection sends carries a Date field, and so does every final response an Http2TlsServer or Http3Server slot sends, as a lowercase date field straight after :status. A 1xx response and trailers carry none. Each server's HttpDate formats the date at most once a second, and a pool's slots share one. Set your own date field and it is sent instead: in any case on HTTP/1.1, and lowercase on HTTP/2 and HTTP/3, which refuse an uppercase field name (H2_INVALID, H3_INVALID). On HTTP/3 the date counts against the client's field-section limit like any other field.

In a test that compares the bytes, call date.pin(unixMs) on the server, or on a lone HTTP/1.1 connection; give a lone Http2Connection or Http3Connection the bytes to send in its date. The formatter itself, httpDateWrite in nish/net/http-fields, reads no clock, so a wasm32 build can use it.

When an Http3Server shuts down, call its stop(): it releases every connection and wipes the stateless reset and Retry token keys of the QuicServerConfig it was made with, so keep no other copy of them, or wipe yours with secureZero too. It zeroes those keys in place, so give each server a QuicServerConfig of its own: once one server stops, any other listener, connection or server made with the same configuration would sign its Retry tokens and stateless resets with a key of zeros, which anyone can forge.

A QuicConnection sends 1,200-byte datagrams until path MTU discovery has validated a larger path, up to 1,452 bytes, and falls back to 1,200 if that path stops carrying them. datagramSize() is the size it sends now, and maxDatagramPayload the largest DATAGRAM it takes. Give takeDatagramInto a buffer of QUIC_CONN_DATAGRAM_MAX bytes to let it probe; QuicListener flights and Http3Server already do. Set pathMtuLimit to what your link carries, or to 1,200 to turn discovery off.

The HTTP/1.1 and WebSocket modules were written with a security record of their own (docs/security/http1-codec.md) rather than audited after the fact. TLS's one-shot key schedule (tlsExtract, tlsDeriveSecret, tlsTrafficKey and the rest of nish/net/tls/schedule) and QUIC's quicKeys, quicKeysUpdate and quicKeyUpdateSecret take a traffic secret as a Secret<u8[]> and answer secrets as one; quicKeys and quicKeysUpdate write the packet keys into a QuicKeysSlot you pass, and that slot is yours to wipe, so call its wipe() (or quicKeysWipe on the keys) when you are done.

A TlsServer keeps every secret of an open connection as a Secret field: the ephemeral key, and the handshake, traffic, exporter and resumption secrets in the TlsSecretPool that secrets() answers, read only through expose. new TlsServer, restart and TlsTcpServer.accept copy the ephemeral key you hand them into the server's own Secret and zero your array, so fill a fresh key into it for each connection rather than reading it again. When you drive a TlsServer or a TlsRecordServer yourself rather than through nish/net/tls-tcp, call its wipe() once the connection is done; the carriers call it at every end of a connection.

When a TlsTcpServer slot ends a connection itself (shutdown, or an HTTP carrier above it after its answer), it does not close the socket at once: it shuts the write side, wipes the slot's keys and answers TLS_RECORD_WANT_READ until the client hangs up, then TLS_RECORD_DONE. Closing a socket the client is still writing to resets the connection, and a browser can report that reset instead of the response. Keep reading the slot until its interest says done, and close slots that stay quiet from your own loop's timeout.

A server that lends byte buffers across connections takes them from nish/net/buffer-pool's BufferPool: a pool of fixed-size buffers, up to a ceiling, whose give zeroes the bytes a buffer used before it is lent again, and whose giveSecret wipes the buffer whole.

A TLS server that resumes sessions seals its tickets with nish/net/tls/ticket: tlsTicketSeal and tlsTicketOpen take the ticket key as a borrowed Secret<u8[]>, so your program keeps the current and previous keys in locals — draw both with tlsTicketKeyDraw at start-up, each into its own TlsTicketKeyInfo — and runs the rotation itself when tlsTicketKeyDue says so — wipe(previous), previous = current, previousInfo.copyFrom(currentInfo), current = tlsTicketKeyDraw(currentInfo, now). Every time it takes is monotonicNanos(), read by you and passed in.

The servers strike each ticket they resume in a TlsTicketStrikes you make and lend them: its take answers false for a replay, and for every ticket once the register is full, so make it as large as the tickets you issue in two hours.

Resumption is off until you turn it on. Call enableTickets() on the TlsServer (conn.tls of a TlsRecordServer) or on the TlsTcpServer, once; it is kept across restart. From then on a connection's interest says TLS_RECORD_TICKET (from nish/net/tls/record-server) twice. In the handshake, when the ClientHello offers a ticket, call resume(current, currentInfo, previous, previousInfo, strikes, now), with the current key as both if you have only one: the server opens the ticket, checks its binder, strikes it and answers the hello, resumed with psk_dhe_ke (no certificate, no signature, so no TLS_RECORD_SIGN) or as a full handshake when the ticket is refused.

Once the connection is open, call issue(current, currentInfo, now) to send it one ticket, whether or not it resumed. On a TlsRecordServer, resume answers 0 or the alert it failed with, and issue whether it sent a ticket; on a TlsTcpServer both take the slot first and answer the slot's new interest bits.

A ticket the server cannot use (an unknown or retired key, an expired or replayed ticket, another suite's hash or another server name, a full register) falls back to a full handshake; a selected ticket whose binder does not verify fails the connection with decrypt_error. QUIC never resumes: enableTickets on a QUIC connection's TlsServer leaves tickets off. A resumed handshake costs the server about 0.6x a full one (tests/link/net_tls_resume_bench).

// In the loop, for a slot `poll` says is readable:
let wants: i32 = tcp.readable(slot);
if ((wants & TLS_RECORD_TICKET) !== 0) {
  wants = tcp.connection(slot).state === TLS_RECORD_STATE_OPEN
    ? tcp.issue(slot, current, currentInfo, monotonicNanos())
    : tcp.resume(slot, current, currentInfo, previous, previousInfo, strikes, monotonicNanos());
}
import { HTTP1_END, HTTP1_ERROR, HTTP1_HEAD, HTTP1_NEED_MORE, Http1Parser } from "nish/net/http1-codec";

const bytesOf = (text: string): u8[] => {
  const out: u8[] = [];
  const n: i32 = toI32(text.length);
  for (let i: i32 = 0; i < n; i++) {
    out.push(toU8(text.charCodeAt(i)));
  }
  return out;
};

export const main = (): i32 => {
  // limits: target bytes, header bytes, header count, body bytes
  const parser = new Http1Parser(1024, 8192, 32, 65536);
  const request = bytesOf("GET /hello HTTP/1.1\r\nHost: example.com\r\n\r\n");
  parser.feed(request, 0, toI32(request.length));
  let event = parser.next();
  while (event !== HTTP1_NEED_MORE && event !== HTTP1_END && event !== HTTP1_ERROR) {
    if (event === HTTP1_HEAD) {
      const host = parser.header("host");
      if (host !== null) {
        console.log(`${parser.method} ${parser.target} for ${host}`);
      }
    }
    event = parser.next();
  }
  return 0;
};
$ nish run request.ts
GET /hello for example.com

nish/net/url

A request target is a path and, after a ?, a query of key=value pairs joined by &, both percent-encoded. nish/net/url reads them. For one value, urlQueryGet(target, name) answers it decoded, "" for a key written without =, or null; urlDecode(s, flags) decodes a whole string. Both allocate their answer and answer null for anything malformed or not UTF-8.

A server reads the bytes it received instead, and allocates nothing: urlQueryStart finds the ?, UrlPathSegments splits the path on the raw /, and UrlQuery yields the pairs one next() at a time, as offsets into your array. percentDecode and percentDecodePathSegment decode one window into a buffer you own and answer how many bytes they wrote. An iterator is made once and reset for each request.

import {
  URL_DECODE_FORM,
  URL_QUERY_PAIR,
  UrlPathSegments,
  UrlQuery,
  UrlQueryLimits,
  percentDecode,
  percentDecodePathSegment,
  urlDecode,
  urlQueryFind,
  urlQueryGet,
  urlQueryStart,
} from "nish/net/url";

const bytesOf = (s: string): u8[] => {
  const out: u8[] = new Array<u8>(toI32(s.length));
  for (let i: i32 = 0; i < toI32(out.length); i++) {
    out[i] = toU8(s.charCodeAt(i));
  }
  return out;
};

const route = (text: string, path: UrlPathSegments, query: UrlQuery): void => {
  const target = bytesOf(text);
  const len = toI32(target.length);
  const out: u8[] = new Array<u8>(len);
  const q = urlQueryStart(target, 0, len); // the `?`, or -1 when there is none
  path.reset(target, 0, q < 0 ? len : q);
  while (path.next()) {
    const n = percentDecodePathSegment(target, path.start, path.length, out, 0, 0);
    if (n < 0) {
      console.log(`bad segment (${n})`); // a URL_DECODE_ERR_* code: answer 400
      return;
    }
    console.log(`segment of ${n} bytes`); // files, a/b (one segment), c
  }
  if (q < 0) {
    console.log("no query");
    return;
  }
  query.reset(target, q + 1, len - q - 1);
  const found = urlQueryFind(query, "name"); // URL_QUERY_PAIR, URL_QUERY_END, or an error code
  if (found !== URL_QUERY_PAIR) {
    console.log(`no name (${found})`);
    return;
  }
  const n = percentDecode(target, query.valueStart, query.valueLength, out, 0, URL_DECODE_FORM);
  if (n < 0) {
    console.log(`bad name (${n})`); // `%zz`: never read out past a negative answer
    return;
  }
  console.log(`name is ${String.fromBytes(out, 0, n)}`); // name is J Doe
};

export const main = (): i32 => {
  // The string API: one call, an allocated answer, null on anything malformed.
  const q0 = urlQueryGet("/search?q=a+b%21&page=2", "q");
  if (q0 !== null) {
    console.log(`q is ${q0}`); // q is a b!
  }
  console.log(`${urlDecode("100%", 0) === null}`); // true: a `%` without two hex digits is refused

  // The byte API: offsets into the target, decoded into a buffer you own.
  const path = new UrlPathSegments();
  const query = new UrlQuery(new UrlQueryLimits()); // at most 100 pairs
  route("/files/a%2Fb/c?name=J%20Doe&flag", path, query);
  route("/files/readme", path, query);
  route("/files/x?name=%zz", path, query);
  route("/files/%zz", path, query);
  return 0;
};
$ nish run route.ts
q is a b!
true
segment of 5 bytes
segment of 3 bytes
segment of 1 bytes
name is J Doe
segment of 5 bytes
segment of 6 bytes
no query
segment of 5 bytes
segment of 1 bytes
bad name (-1)
segment of 5 bytes
bad segment (-1)

Decoding is strict by default. With flags 0, percentDecode answers URL_DECODE_ERR_ESCAPE for a % not followed by two hex digits and URL_DECODE_ERR_NUL for %00, and copies + as it is. Add the flags you want: URL_DECODE_FORM reads + as a space, as a form or a query does; URL_DECODE_LENIENT copies a malformed escape as it is; URL_DECODE_ALLOW_NUL decodes %00 to a 0 byte. All three together decode as URLSearchParams does for valid UTF-8; where it replaces an invalid sequence with U+FFFD, percentDecode keeps the bytes.

A buffer as long as the input is always big enough; one shorter than the decoded result answers URL_DECODE_ERR_SPACE, with the bytes before it written. Test every answer before you read out: a peer's %zz makes it negative, and String.fromBytes(out, 0, n) with a negative n panics. percentDecodeErrorAt, called with the same arguments, answers the offset of the byte an error is at, or -1.

A path is split before it is decoded, so %2F stays inside its segment: /a%2Fb/c is the two segments a/b and c, never three. + is literal in a path.

A query yields at most 100 pairs. Past UrlQueryLimits.maxPairs, next() answers URL_QUERY_ERR_PAIRS, and keeps answering it until reset, so a target of a hundred thousand &-joined pairs costs one bounded scan. Empty pairs (a&&b) are skipped and not counted. urlQueryGet uses the default cap, and answers null for a key past it.

The byte API does not check that what it decodes is UTF-8: a value may be binary. The name you give urlQueryFind or urlQueryGet is ASCII: the native build compares it with the key's bytes and Node with its UTF-16 code units, so a non-ASCII name answers differently on the two. The byte API never looks for a #, which is never part of a request target.

Whole URLs

A client handed a URL by a user, a redirect or a link needs its parts: urlParse(s) splits an absolute URL into a Url (scheme, username, password, host, port, path, query, fragment), and urlResolve(base, ref) resolves a reference against one, as a browser does for a link, removing . and .. so a path never climbs above /. urlSerialize writes a Url back.

The scheme and host are lower-cased; everything else is kept as written, still percent-encoded. port is -1 when the URL names none; urlDefaultPort(scheme) is 80 or 443, and urlOrigin(u) writes scheme://host:port with the port always there, so origins compare as plain strings. A host is compared as written, so http://a.com./ (a trailing dot) and http://[0::1]/ are different origins from http://a.com/ and http://[::1]/: an equal string is the same origin, and a split errs toward "different".

import { Url, urlOrigin, urlParse, urlResolve, urlSerialize } from "nish/net/url";

const show = (base: Url, ref: string): void => {
  const u: Url | null = urlResolve(base, ref);
  if (u === null) {
    console.log(`${ref} is refused`);
    return;
  }
  console.log(`${ref} -> ${urlSerialize(u)}, host ${u.host}, port ${u.port}`);
};

export const main = (): i32 => {
  const base: Url | null = urlParse("https://Example.com/docs/guide/index.html?v=2");
  if (base === null) {
    return 1;
  }
  show(base, "../api/url.html#parse");
  show(base, "/../../etc/passwd");
  show(base, "?v=3");
  show(base, "//cdn.example.com:8443/app.js");
  show(base, "http://127.1/");
  show(base, "http://good.com%2f@evil.com/");
  show(base, "http://a@b@c/");
  const origin: string | null = urlOrigin(base);
  console.log(origin === null ? "no origin" : origin);
  return 0;
};
$ nish run links.ts
../api/url.html#parse -> https://example.com/docs/api/url.html#parse, host example.com, port -1
/../../etc/passwd -> https://example.com/etc/passwd, host example.com, port -1
?v=3 -> https://example.com/docs/guide/index.html?v=3, host example.com, port -1
//cdn.example.com:8443/app.js -> https://cdn.example.com:8443/app.js, host cdn.example.com, port 8443
http://127.1/ is refused
http://good.com%2f@evil.com/ -> http://good.com%2f@evil.com/, host evil.com, port -1
http://a@b@c/ is refused
https://example.com:443

A URL two readers could disagree on is refused, with null, not repaired: a byte outside printable ASCII (a space, a tab, a non-ASCII letter), a backslash anywhere (a browser reads it as /), a % not followed by two hex digits, a second @, a host byte other than a letter, a digit and -._~ (so a percent-encoded byte), an IPv6 zone, a port past 65535, and a host that looks like a number but is not four decimal parts (127.1, 0x7f.0.0.1, 2130706433, 010.0.0.1).

An http, https, ws or wss URL needs a host. What does parse is always one host: in http://good.com%2f@evil.com/, everything before the @ is the username.

Resolution is RFC 3986's, strictly: http:g against an http base is an absolute URL with no host, so it is refused, where a browser reads it as the relative g. A non-ASCII host is refused, so pass an international name in its punycode xn-- form. An IPv6 host is compared as written, so [::1] and [0::1] are different origins; check an address after you resolve the name, never the string, before you trust it.

nish/net/dns-message

A DNS lookup is one UDP datagram out and one back. nish/net/dns-message writes the first and reads the second; it opens no socket, so you send and receive the bytes yourself. dnsBuildQuery(out, id, name, qtype) writes a question for name of type DNS_TYPE_A or DNS_TYPE_AAAA, with recursion asked for, and answers its length: out of DNS_QUERY_MAX_BYTES (271) always has room. A name that is not one (an empty label, a label over 63 bytes, a name over 253, a byte outside printable ASCII) answers a negative DNS_ERR_* instead. Pick id at random for each query: it is half of what keeps a forged reply out.

dnsParseReply(msg, len, query, queryLen, port, reply) reads a reply into a DnsReply you make once and reuse. It answers the reply's RCODE (DNS_NOERROR, DNS_NXDOMAIN, DNS_SERVFAIL, DNS_REFUSED) or a negative DNS_ERR_*: a reply whose ID, question or flags do not match your query is refused before a record is read, so read it as "not my answer" and keep waiting. On DNS_NOERROR, reply.count addresses are in reply.addresses, 18 bytes each, the form nish:net connects to, with port written in; at most DNS_MAX_ANSWERS (8) are kept and reply.dropped counts the rest. When reply.truncated is set, nothing was read: ask again over TCP.

import {
  DNS_ERR_NOT_REPLY,
  DNS_QUERY_MAX_BYTES,
  DNS_TYPE_A,
  DnsReply,
  dnsBuildQuery,
  dnsParseReply,
} from "nish/net/dns-message";

export const main = (): i32 => {
  const query: u8[] = new Array<u8>(DNS_QUERY_MAX_BYTES);
  const n: i32 = dnsBuildQuery(query, 0x1234, "example.com", DNS_TYPE_A);
  console.log(`${n} bytes`);
  // A reply that is only an echo of the query has QR clear: not a reply.
  const reply = new DnsReply();
  const status: i32 = dnsParseReply(query, n, query, n, 443, reply);
  console.log(`${status === DNS_ERR_NOT_REPLY ? "not a reply" : "read"}`);
  return 0;
};
$ nish run lookup.ts
29 bytes
not a reply

A CNAME chain is followed inside the one reply. An address is kept only when its owner is the name you asked, or the last name a CNAME record before it pointed to; a record for any other name is read past. Compression pointers must point backwards, at most 16 deep, and a name never grows past 255 bytes, so no reply makes the parser loop or read outside msg[0, len).

nish/net/dns-config

nish/net/dns-config reads /etc/resolv.conf and /etc/hosts from bytes you have already read, into tables of fixed size. dnsParseResolvConf(b, off, len, config) fills a DnsConfig: up to three servers (18 bytes each, on port 53), up to six search domains in 256 bytes, and ndots, timeout and attempts from options, each clamped (0 to 15, 1 to 30, 1 to 5). A file with no usable nameserver gives 127.0.0.1 and ::1, as glibc does. dnsParseHosts(b, off, len, hosts) fills a DnsHosts, and dnsHostsFind(hosts, name, qtype, port, out, max) writes the addresses it gives name, without case, in file order, and answers how many.

import { DnsHosts, dnsHostsFind, dnsParseHosts } from "nish/net/dns-config";
import { DNS_TYPE_A } from "nish/net/dns-message";

const bytesOf = (text: string): u8[] => {
  const out: u8[] = [];
  const n: i32 = toI32(text.length);
  for (let i: i32 = 0; i < n; i++) {
    out.push(toU8(text.charCodeAt(i)));
  }
  return out;
};

export const main = (): i32 => {
  const file = bytesOf("127.0.0.1 localhost\n::1 localhost\n192.0.2.7 Build.Lan build # CI\n");
  const hosts = new DnsHosts();
  dnsParseHosts(file, 0, toI32(file.length), hosts);
  const out: u8[] = new Array<u8>(4 * 18);
  console.log(`${dnsHostsFind(hosts, "BUILD", DNS_TYPE_A, 22, out, 4)} address, ${out[15]}`);
  return 0;
};
$ nish run hosts.ts
1 address, 7

Neither file is trusted to be small: what does not fit a table is dropped and counted (droppedServers, droppedSearch, droppedEntries, droppedNames), and a line over 1024 bytes is skipped whole and counted in skippedLines. An address is a strict literal: 010.0.0.1 and fe80::1%eth0 are refused, not read as octal or with a zone. dnsParseAddress is the same reader, for a literal anywhere in your own bytes.

nish/net/dns-resolver

DnsResolver is a stub resolver: it asks the servers in a DnsConfig for a name's A and AAAA records and gives you the addresses. It opens no socket, reads no clock and reads no file, so it builds for wasm32 too; you, or nish/net/dns, move its bytes. new DnsResolver(config, hosts, maxLookups) makes every buffer it will need, for up to maxLookups lookups at once (1 to 64), so a lookup allocates only the sixteen bytes an address literal is parsed into.

start(name, port, nowNanos) begins a lookup and answers its slot. nextOutgoing(out, to, meta, id) says what to do next, a query to send over UDP or TCP or a socket to close, with id a fresh random 16-bit number each time; receive and receiveStream take what came back; tick(nowNanos) moves on from a server that did not answer by nextDeadlineNanos(). result(lookup, out, ttls) answers DNS_RESOLVE_PENDING while the lookup runs, then the address count, A first, 18 bytes each with port written in, and each TTL in seconds: the smallest TTL of the address records, not of the CNAMEs that led to them.

A failure is DNS_RESOLVE_NOT_FOUND (the name has no address: NXDOMAIN, or NODATA, a name with no A or AAAA record), DNS_RESOLVE_NO_ANSWER (no server answered) or DNS_RESOLVE_CNAME (an alias chain ran past 8, or led back to the name asked across replies; a loop inside one reply is DNS_RESOLVE_NOT_FOUND). start itself answers a negative code when it cannot begin: DNS_RESOLVE_BUSY when every slot is taken, or the DNS_ERR_* code of nish/net/dns-message for a name that is not one, such as "" or a label over 63 bytes. Check it before you pass it to result, which panics on a number that is not a lookup.

localhost, an address literal and a name your hosts table has answer at once, with no packet sent:

import { DnsResolver } from "nish/net/dns-resolver";
import { DnsConfig, dnsParseResolvConf } from "nish/net/dns-config";
import { DNS_ADDRESS_BYTES } from "nish/net/dns-message";

export const main = (): i32 => {
  const config = new DnsConfig();
  dnsParseResolvConf(new Array<u8>(0), 0, 0, config);
  const resolver = new DnsResolver(config, null, 1);
  const lookup: i32 = resolver.start("localhost", 443, toI64(0));
  const out: u8[] = new Array<u8>(16 * DNS_ADDRESS_BYTES);
  const ttls: i32[] = new Array<i32>(16);
  console.log(`${resolver.result(lookup, out, ttls)} addresses, 127.0.0.${out[15]} first`);
  return 0;
};
$ nish run local.ts
2 addresses, 127.0.0.1 first

A forged reply cannot end a lookup. A reply from another address or port than the server's, or with another ID or question, is dropped and counted (droppedSource, droppedId, droppedQuestion), and the lookup keeps waiting for the true one.

A truncated reply is asked again over TCP, once; a server that does not answer in timeout seconds gives way to the next, for attempts rounds; a name with fewer dots than ndots is tried with each search domain. When one family answers NXDOMAIN and the other reached no server, the next candidate is tried, and if none answers the lookup ends with the unreachable family's error. Every counter stops at 2,147,483,647 rather than wrapping. Every lookup ends.

nish/net/dns

nish/net/dns drives a DnsResolver over nish:net sockets on the loop your program already runs, so a lookup never stops a server from serving. new DnsClient(resolver, loop, tokenBase) joins your pollCreate loop under the tokens tokenBase to tokenBase + client.tokens() - 1.

Start a lookup with client.start(name, port, now), pass client.timeoutMs(now, yours) to pollWait, hand every ready token client.owns(token) claims to client.ready(token, events, now), call client.tick(now) after each wake, and read client.result(lookup, out, ttls) until it is not DNS_RESOLVE_PENDING. Each query goes out on a fresh socket with a random ID, and one ready reads at most DNS_CLIENT_MAX_DATAGRAMS (32) datagrams, so a flood on the port cannot keep your loop inside the client.

import { DnsClient } from "nish/net/dns";
import { DNS_RESOLVE_PENDING, DnsResolver } from "nish/net/dns-resolver";
import { DnsConfig, dnsParseResolvConf } from "nish/net/dns-config";
import { DNS_ADDRESS_BYTES } from "nish/net/dns-message";
import { pollCreate, pollWait } from "nish:net";
import { monotonicNanos } from "nish:process";

export const main = (): i32 => {
  const config = new DnsConfig();
  const conf: u8[] = [];
  const text = "nameserver 192.0.2.53\n";
  for (let i: i32 = 0; i < toI32(text.length); i++) {
    conf.push(toU8(text.charCodeAt(i)));
  }
  dnsParseResolvConf(conf, 0, toI32(conf.length), config);
  const loop: i32 = pollCreate();
  const client = new DnsClient(new DnsResolver(config, null, 4), loop, 100);
  const lookup: i32 = client.start("example.com", 443, monotonicNanos());
  if (lookup < 0) {
    console.log(`not started: ${lookup}`);
    return 1;
  }
  const out: u8[] = new Array<u8>(16 * DNS_ADDRESS_BYTES);
  const ttls: i32[] = new Array<i32>(16);
  const ready: i32[] = new Array<i32>(16);
  let status: i32 = client.result(lookup, out, ttls);
  while (status === DNS_RESOLVE_PENDING) {
    // Your own sockets share this loop; -1 means you have no timer of your own.
    const count: i32 = pollWait(loop, ready, client.timeoutMs(monotonicNanos(), -1));
    const now: i64 = monotonicNanos();
    for (let k: i32 = 0; k < count; k++) {
      if (client.owns(ready[2 * k])) {
        client.ready(ready[2 * k], ready[2 * k + 1], now);
      }
    }
    client.tick(now);
    status = client.result(lookup, out, ttls);
  }
  console.log(`${status}`);
  return 0;
};

resolveHost(name, timeoutMs, port, out, ttls) is the same lookup with /etc/resolv.conf and /etc/hosts read for you, on a loop of its own: it blocks until the lookup ends or timeoutMs passes (DNS_RESOLVE_TIMEOUT), so it is for a command-line tool or a program's start-up, not a server's loop. It waits for both the A and the AAAA answer until timeoutMs passes; then a family that answered with addresses is returned on its own, and only a lookup with none yet is DNS_RESOLVE_TIMEOUT. A pollWait that fails cancels the lookup and its negative errno is the answer; a name start refuses answers start's code. dnsLookupWith is the same with a configuration you give it.

import { resolveHost } from "nish/net/dns";
import { DNS_ADDRESS_BYTES } from "nish/net/dns-message";

export const main = (): i32 => {
  const out: u8[] = new Array<u8>(16 * DNS_ADDRESS_BYTES);
  const ttls: i32[] = new Array<i32>(16);
  console.log(`${resolveHost("localhost", 2000, 80, out, ttls)} addresses`);
  return 0;
};
$ nish run host.ts
2 addresses

A program that uses DnsClient reports net and entropy under --emit-capabilities, the IDs being random; resolveHost adds fs.read for the two files and clock for its deadline. Both are native only, as every nish:net call is.

The library's README lists every module, every export and the specification each one reproduces. How nish/ imports resolve is in export and import, and Modules covers importing your own files.