nish/bytes

Fixed-width integers and doubles read from, and written into, a byte array at an offset, in either byte order.

import {
  copyBytes,
  getF64BE,
  getF64LE,
  getI32BE,
  getI32LE,
  getI64BE,
  getI64LE,
  getU16BE,
  getU16LE,
  getU32BE,
  getU32LE,
  getU64BE,
  getU64LE,
  setF64BE,
  setF64LE,
  setI32BE,
  setI32LE,
  setI64BE,
  setI64LE,
  setU16BE,
  setU16LE,
  setU32BE,
  setU32LE,
  setU64BE,
  setU64LE,
  skipBlank,
  skipDigits,
  skipJsonPlain,
  skipJsonText,
} from "nish/bytes";
import { getU32BE, setU16LE } from "nish/bytes";

export const main = (): i32 => {
  const header: u8[] = [0, 0, 1, 0, 0, 0];
  const length = getU32BE(header, 0); // 256
  setU16LE(header, 4, toU16(513)); // header[4] = 1, header[5] = 2
  return toI32(length);
};

A wire format, a file header or a hash's message schedule is a u8[] with numbers laid out in it, and every reader of one writes the same four-line shift-and-or. This module is that loop written once:

Every function takes the array and the offset of its first byte; the LE name puts the least significant byte there, the BE name the most significant. The widths are u16, u32 and u64, the signed i32 and i64, which are the same bytes read as two's complement, and f64, whose bytes are its IEEE 754 bits (f64ToBits). There is no f32: the language has no bit cast for one.

Each access is an ordinary index

Each access is an ordinary index, so an offset whose bytes do not all lie in the array panics index, naming the first byte that does not.

A guard in the caller does not yet remove these checks: in a loop guarded by off + 3 < b.length, the checker proves an index it can see but not one inside a call, and after getU32LE inlines, LLVM keeps the checks on off + 1 to off + 3, because the guard compares the length truncated to i32 and the checks compare it unsigned at 64 bits. A loop that needs them gone indexes the bytes itself.

getU64BE, and with it getI64BE and getF64BE, is the exception: it checks the whole range once and reads the eight bytes unchecked, which lowers to one load and a byte swap. An offset outside the range still goes byte by byte, so it panics as the others do.

The in-range answers are byte for byte those of the copies std/ keeps for itself: sha256LoadWord (std/crypto/sha256.ts) for getU32BE, chacha20Word and poly1305Le64 (std/crypto/chacha20poly1305.ts) for getU32LE and getU64LE, loadWord and storeWord (std/crypto/sha512.ts) for getU64BE and setU64BE, http2FramePut32 (std/net/http2-frame.ts) for setU32BE, and tlsPutU16 (std/net/tls/codec.ts) for setU16BE. None of them is replaced by this module. Out of range they differ: the AES and TLS word helpers read zero past the end and drop bytes written there, which their callers rely on, where this module panics.

Under Node (docs/RUN_UNDER_NODE.md) the u16 and u32 functions are expected to answer as they do natively, because the top byte of a u32 is multiplied in rather than shifted and each byte written is masked before its toU8: there << turns bit 31 into a sign and the prelude's toU8 and toU32 saturate, and natively the multiply and the mask fold into the shift and the truncation. The others do not yet answer there: the i32 functions cross between signed and unsigned with toI32 and toU32, which the prelude saturates too (setI32LE(b, 0, -2) writes zeros), and the 64-bit functions and f64 meet its BigInt u64 and i64, which toU8 refuses.

The module also has the runs a parser walks and the copy it makes of them. skipBlank, skipDigits, skipJsonText and skipJsonPlain each answer the first index in a range whose byte is not in their class — JSON's blanks, the ASCII digits, the bytes a JSON string holds as themselves, the bytes a JSON writer copies without an escape — and copyBytes copies a window of one array into another, or into itself, as one move:

import { skipBlank, skipDigits } from "nish/bytes";

export const main = (): i32 => {
  const text: u8[] = [32, 32, 52, 50, 44]; // "  42,"
  const start = skipBlank(text, 0, 5); // 2
  const end = skipDigits(text, start, 5); // 4
  return end - start;
};

The four runs clamp their range and never panic; copyBytes panics on a range outside its array. Natively each is a runtime kernel (docs/LANGUAGE.md, "Byte runs"), and the runs are variable-time: they are for text, never for secret data.

Functions

copyBytes

count bytes of src from from written over dst from at, as one copy: where dst and src are one array and the two ranges overlap, the bytes land as if every one were read before any was written, as copyWithin promises. Where dst.set(src, at) copies all of src, this copies a window of it, and allocates nothing.

A negative count, or a range that does not lie inside its array — 0 <= from <= from + count <= src.length and the same for at in dst — panics copyBytes before any byte is written. Natively the copy is one llvm.memmove.

const copyBytes: (
  dst: u8[],
  at: i32,
  src: readonly u8[],
  from: i32,
  count: i32,
) => void
dstu8[]required

The array written into; only dst[at..at+count) changes.

ati32required

Where in dst the first copied byte goes.

srcreadonly u8[]required

The array copied from, which may be dst itself.

fromi32required

Where in src the copied bytes start.

counti32required

How many bytes to copy; zero copies nothing.

getF64BE

The f64 whose IEEE 754 bits are getU64BE's.

const getF64BE: (b: u8[], off: i32) => f64
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

returnsf64

The double whose bits the eight bytes are.

getF64LE

The f64 whose IEEE 754 bits are getU64LE's.

const getF64LE: (b: u8[], off: i32) => f64
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

returnsf64

The double whose bits the eight bytes are.

getI32BE

getU32BE's bits as a two's-complement i32.

const getI32BE: (b: u8[], off: i32) => i32
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

returnsi32

The number read, from -2^31 to 2^31 - 1.

getI32LE

getU32LE's bits as a two's-complement i32.

const getI32LE: (b: u8[], off: i32) => i32
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

returnsi32

The number read, from -2^31 to 2^31 - 1.

getI64BE

getU64BE's bits as a two's-complement i64.

const getI64BE: (b: u8[], off: i32) => i64
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

returnsi64

The number read, from -2^63 to 2^63 - 1.

getI64LE

getU64LE's bits as a two's-complement i64.

const getI64LE: (b: u8[], off: i32) => i64
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

returnsi64

The number read, from -2^63 to 2^63 - 1.

getU16BE

The u16 whose high byte is b[off] and low byte b[off + 1].

const getU16BE: (b: u8[], off: i32) => u16
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 1], and a byte of those outside b panics index.

returnsu16

The number read.

getU16LE

The u16 whose low byte is b[off] and high byte b[off + 1].

const getU16LE: (b: u8[], off: i32) => u16
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 1], and a byte of those outside b panics index.

returnsu16

The number read.

getU32BE

The u32 whose bytes, most significant first, are b[off] to b[off + 3].

const getU32BE: (b: u8[], off: i32) => u32
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

returnsu32

The number read.

getU32LE

The u32 whose bytes, least significant first, are b[off] to b[off + 3].

const getU32LE: (b: u8[], off: i32) => u32
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

returnsu32

The number read.

getU64BE

The u64 whose bytes, most significant first, are b[off] to b[off + 7]. The fast path's ors are grouped in pairs, which LLVM folds into one load and a bswap; a flat chain of eight ors stayed byte by byte. The byte loop is its own function so its last load is not merged with the fast path's.

const getU64BE: (b: u8[], off: i32) => u64
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

returnsu64

The number read.

getU64LE

The u64 whose bytes, least significant first, are b[off] to b[off + 7].

const getU64LE: (b: u8[], off: i32) => u64
bu8[]required

The bytes the number is read from; they are not changed.

offi32required

Where in b its first byte is; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

returnsu64

The number read.

setF64BE

Writes v's IEEE 754 bits as setU64BE does.

const setF64BE: (b: u8[], off: i32, v: f64) => void
bu8[]required

The bytes written into; only the eight at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

vf64required

The double to store, as its IEEE 754 bits.

setF64LE

Writes v's IEEE 754 bits as setU64LE does.

const setF64LE: (b: u8[], off: i32, v: f64) => void
bu8[]required

The bytes written into; only the eight at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

vf64required

The double to store, as its IEEE 754 bits.

setI32BE

Writes v's two's-complement bits as setU32BE does.

const setI32BE: (b: u8[], off: i32, v: i32) => void
bu8[]required

The bytes written into; only the four at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

vi32required

The number to store; a negative one is stored as its two's complement.

setI32LE

Writes v's two's-complement bits as setU32LE does.

const setI32LE: (b: u8[], off: i32, v: i32) => void
bu8[]required

The bytes written into; only the four at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

vi32required

The number to store; a negative one is stored as its two's complement.

setI64BE

Writes v's two's-complement bits as setU64BE does.

const setI64BE: (b: u8[], off: i32, v: i64) => void
bu8[]required

The bytes written into; only the eight at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

vi64required

The number to store; a negative one is stored as its two's complement.

setI64LE

Writes v's two's-complement bits as setU64LE does.

const setI64LE: (b: u8[], off: i32, v: i64) => void
bu8[]required

The bytes written into; only the eight at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

vi64required

The number to store; a negative one is stored as its two's complement.

setU16BE

Writes v into b[off] and b[off + 1], high byte first.

const setU16BE: (b: u8[], off: i32, v: u16) => void
bu8[]required

The bytes written into; only the two at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 1], and a byte of those outside b panics index.

vu16required

The number to store.

setU16LE

Writes v into b[off] and b[off + 1], low byte first.

const setU16LE: (b: u8[], off: i32, v: u16) => void
bu8[]required

The bytes written into; only the two at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 1], and a byte of those outside b panics index.

vu16required

The number to store.

setU32BE

Writes v into b[off] to b[off + 3], most significant byte first.

const setU32BE: (b: u8[], off: i32, v: u32) => void
bu8[]required

The bytes written into; only the four at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

vu32required

The number to store.

setU32LE

Writes v into b[off] to b[off + 3], least significant byte first.

const setU32LE: (b: u8[], off: i32, v: u32) => void
bu8[]required

The bytes written into; only the four at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 3], and a byte of those outside b panics index.

vu32required

The number to store.

setU64BE

Writes v into b[off] to b[off + 7], most significant byte first.

const setU64BE: (b: u8[], off: i32, v: u64) => void
bu8[]required

The bytes written into; only the eight at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

vu64required

The number to store.

setU64LE

Writes v into b[off] to b[off + 7], least significant byte first.

const setU64LE: (b: u8[], off: i32, v: u64) => void
bu8[]required

The bytes written into; only the eight at off change.

offi32required

Where in b the first byte goes; it covers b[off] to b[off + 7], and a byte of those outside b panics index.

vu64required

The number to store.

skipBlank

The first index in [from, to) whose byte is not one of JSON's blanks — space, tab, newline, carriage return — or to when every byte there is one. to is clamped to [0, b.length] and from to [0, to], as indexOfAny clamps, so no range panics: one past the end answers the end.

Natively the scan is the runtime's nish_bytes_skip_blank, which reads 16 or 32 bytes at a time (docs/LANGUAGE.md, "Byte runs"). It is variable-time: how long it takes says how long the run is, so it is never for secret data.

const skipBlank: (b: u8[], from: i32, to: i32) => i32
bu8[]required

The bytes scanned; they are not changed.

fromi32required

Where the scan starts, clamped into [0, to].

toi32required

Where the scan stops, exclusive, clamped into [0, b.length].

returnsi32

The index of the first byte in the clamped range that is not a blank, or the clamped to when there is none.

skipDigits

skipBlank for the ASCII digits 0 to 9: the same clamps, the same kernel family, nish_bytes_skip_digits.

const skipDigits: (b: u8[], from: i32, to: i32) => i32
bu8[]required

The bytes scanned; they are not changed.

fromi32required

Where the scan starts, clamped into [0, to].

toi32required

Where the scan stops, exclusive, clamped into [0, b.length].

returnsi32

The index of the first byte in the clamped range that is not a digit, or the clamped to when there is none.

skipJsonPlain

The first index in [from, to) whose byte a JSON writer cannot copy as itself — a control byte below 0x20, a quote or a backslash — or to, with the range clamped as skipBlank's is. Every byte from 0x20 up is in the run, 0x7f and the bytes from 0x80 up included, and no UTF-8 is checked: where skipJsonText is the reader's run, this is the writer's, which passes the bytes of a string through as they are. Natively the scan is nish_bytes_skip_json_plain, 16 bytes at a time. Variable-time, as skipBlank is.

const skipJsonPlain: (b: u8[], from: i32, to: i32) => i32
bu8[]required

The bytes scanned; they are not changed.

fromi32required

Where the scan starts, clamped into [0, to].

toi32required

Where the scan stops, exclusive, clamped into [0, b.length].

returnsi32

The index of the first byte in the clamped range that a JSON writer must escape, or the clamped to when there is none.

skipJsonText

skipBlank for the bytes a JSON string holds as themselves: printable ASCII other than " and \, and every well-formed UTF-8 sequence (RFC 3629 §4) that ends before to. The run stops at the first quote, backslash or control byte, and at the lead byte of a sequence that is malformed — an overlong form, a surrogate, a code point past U+10FFFF, a stray continuation byte — or cut short by to.

So a whole JSON string body with no escape is one call, and where it stops is the byte a decoder has to look at. Natively the scan is nish_bytes_skip_json_text: blocks of ASCII 16 or 32 bytes at a time, each multi-byte sequence a byte at a time. Variable-time, as skipBlank is.

const skipJsonText: (b: u8[], from: i32, to: i32) => i32
bu8[]required

The bytes scanned; they are not changed.

fromi32required

Where the scan starts, clamped into [0, to].

toi32required

Where the scan stops, exclusive, clamped into [0, b.length].

returnsi32

The index of the first byte in the clamped range that is not in the run, or the clamped to when there is none.