nish/text

The string operations a program has to write itself.

import {
  asciiLower,
  asciiUpper,
  codePointLength,
  contains,
  firstDifference,
  indexOfAny,
  padEnd,
  padStart,
  replaceAll,
  split,
  splitLimit,
  splitLines,
  splitWhitespace,
  toFixed,
  trim,
  trimEnd,
  trimStart,
} from "nish/text";
import { asciiUpper, padStart, replaceAll, split, splitLines, splitWhitespace, trim } from "nish/text";

export const main = (): i32 => {
  const lines: string[] = splitLines("a b\n  c  \n"); // ["a b", "  c  "]
  const words: string[] = splitWhitespace(lines[0]); // ["a", "b"]
  const c: string = trim(lines[1]); // "c"
  const dashed: string = replaceAll("a b", " ", "-"); // "a-b"
  const fields: string[] = split("a,,b", ","); // ["a", "", "b"]
  const loud: string = asciiUpper("get_id"); // "GET_ID"
  const code: string = padStart("7", 3, "0"); // "007"
  return toI32(words.length);
};

The language gives a string length, charCodeAt, substring, indexOf, startsWith, endsWith and === by content, and an array join. It does not give split, trim, toLowerCase or a regular expression (that is std/regex), and that is a decision rather than a gap: each allocates, and several need a locale or a character table this runtime has no room for (docs/wp7-runtime.md keeps the runtime under a measured byte budget).

So a program that reads a file and wants its lines writes the loop

This module is that loop, written once. Every offset here is a byte offset, like s.length itself, and every function but one is ASCII-only where it inspects characters at all — a UTF-8 string passes through splitLines, splitWhitespace, asciiUpper and asciiLower untouched, because an ASCII byte cannot appear inside a multi-byte sequence. The one is codePointLength, which counts code points rather than bytes and answers the same natively and under Node; it says how. padStart and padEnd count a width in s.length's units, so they are the two whose answers differ under Node on text that is not ASCII; each says by how much.

Every width this module declares is spelled

Every width this module declares is spelled (i32, never number) and every value a builtin hands it is converted at the read: s.length, a.length, s.charCodeAt(i) and s.indexOf(t) all answer number (LANGUAGE.md, Arrays and strings as receivers), and number is f64 under --number-mode f64. Spelling the declarations is therefore necessary and not sufficient — without the toI32 at each read this module does not compile in that mode at all (docs/wp26-stdlib.md §4). A length a loop tests on every iteration is read into an i32 local once rather than converted per iteration, which is both cheaper and how the loop wants to read; a string is immutable, so its length cannot change underneath the local.

Functions

asciiLower

text with its ASCII letters A to Z made a to z, and nothing else changed: every other byte, a UTF-8 sequence's included, passes through.

This is not JavaScript's toLowerCase, for the reason asciiUpper gives: "É" stays "É" here. On ASCII text the two agree, and this answers the same natively and under Node for any text.

const asciiLower: (text: string) => string
textstringrequired

The text whose ASCII letters are lower-cased.

returnsstring

A fresh string as long as text.

asciiUpper

text with its ASCII letters a to z made A to Z, and nothing else changed: every other byte, a UTF-8 sequence's included, passes through.

This is not JavaScript's toUpperCase, which knows Unicode's case table: that gives "SS" for "ß" and "É" for "é", where this gives "ß" and "é". That table is more than this runtime has room for, and the text this is for, identifiers, keywords and protocol tokens, is ASCII. On ASCII text the two agree, and this answers the same natively and under Node for any text, since a byte or unit from 97 to 122 is always the letter itself, so NL8001 (--warn-portability) has nothing to warn about, and does not.

const asciiUpper: (text: string) => string
textstringrequired

The text whose ASCII letters are upper-cased.

returnsstring

A fresh string as long as text.

codePointLength

The number of Unicode code points in text: what JSON Schema's minLength and maxLength count, and not what text.length counts. codePointLength("é") is 1 where "é".length is 2, and codePointLength("😀") is 1 where its length is 4.

Natively a string is its UTF-8 bytes, and every code point has exactly one byte that is not a continuation byte (10xxxxxx), so that is what is counted. Under Node the same charCodeAt reads UTF-16 units, where a byte rule would miscount: "©®±" is three units in 0x80–0xBF, which are continuation bytes in UTF-8 but whole characters here. So the function asks which host it is on (textUnitsAreBytes) and, under Node, counts every unit except the low half of a surrogate pair.

Both walks answer the same for any string literal, a lone surrogate escape included: it is three WTF-8 bytes with one leading byte natively, and one unit under Node. They differ for text built from bytes: String.fromBytes and String.fromCharCode give one unit per byte under Node, so a string decoded that way, as nish/json decodes a \u escape, counts its UTF-8 bytes there. They also differ for text that is not well-formed — bytes that are not UTF-8 natively, where each stray continuation byte counts nothing, or the two halves of a pair joined at run time, which are one string under Node and two lone surrogates natively.

const codePointLength: (text: string) => i32
textstringrequired

The text whose code points are counted.

returnsi32

How many code points text holds, counted as above.

contains

Whether needle occurs anywhere in haystack. One indexOf, named, because indexOf(...) >= 0 at a call site reads as arithmetic where the question is a yes or a no. contains(s, "") is true, as indexOf("") is 0.

const contains: (
  haystack: string,
  needle: string,
) => boolean
haystackstringrequired

The text searched.

needlestringrequired

The text looked for, compared byte for byte.

returnsboolean

true when needle occurs in haystack.

firstDifference

The index of the first line at which left and right differ, or -1 when one is a prefix of the other and they have the same length — that is, when they are equal. A caller comparing generated text against a golden wants the first differing line and not a boolean, because printing two whole files is not a diagnostic.

const firstDifference: (
  left: string[],
  right: string[],
) => i32
leftstring[]required

One side's lines.

rightstring[]required

The other side's lines, compared to left's with !== by content.

returnsi32

The index of the first line that differs, the shorter length when one side is a prefix of the other, or -1 when they are equal.

indexOfAny

The first index at or after from whose byte is one of bytes, or -1.

bytes is a set of 1 to 16 ASCII bytes, and anything else panics: an empty set matches nothing, more than 16 is more than one vector of the runtime kernel compares, and an ASCII set is what keeps the answer the same under Node, where charCodeAt reads a UTF-16 unit rather than a byte. An ASCII byte never occurs inside a multi-byte UTF-8 sequence, so a non-ASCII text passes through and only its ASCII bytes can match — as in splitLines, the index is a byte offset natively. from is clamped to [0, text.length], as substring clamps, so a negative from searches from the start and one past the end answers -1.

Natively the search is the runtime's nish_str_index_of_any, which reads 16 or 32 bytes at a time (docs/LANGUAGE.md, "Byte search"). It is variable-time: how long it takes says where the first match is, so it is never for secret data.

const indexOfAny: (
  text: string,
  bytes: string,
  from: i32,
) => i32
textstringrequired

The text searched.

bytesstringrequired

The set of bytes looked for, written as a string of 1 to 16 ASCII characters; their order does not matter.

fromi32required

Where the search starts, as a byte offset, clamped into [0, text.length].

returnsi32

The byte offset of the first match at or after from, or -1 when there is none.

padEnd

text with fill repeated after it until it is width units long: JavaScript's text.padEnd(width, fill) for ASCII text, with padStart's rules. padEnd("7", 4, "ab") is "7aba", and padEnd("é", 3, " ") is "é " natively and "é " under Node.

const padEnd: (
  text: string,
  width: i32,
  fill: string,
) => string
textstringrequired

The text to pad.

widthi32required

The length the answer is padded to, in text.length's units.

fillstringrequired

The ASCII text repeated as the padding.

returnsstring

text with the padding after it; text unchanged when no padding is needed.

padStart

text with fill repeated in front of it until it is width units long: JavaScript's text.padStart(width, fill) for ASCII text. A width no longer than text, or an empty fill, gives text unchanged, and a fill of several characters repeats and is cut at the width, so padStart("7", 4, "ab") is "aba7".

width counts the units text.length counts: bytes natively and UTF-16 units under Node. So on text that is not ASCII the two answers differ: padStart("é", 3, " ") is " é" natively, where é is two bytes, and " é" under Node, where it is one unit. NL8001 (--warn-portability) does not warn about it: the warning reads offsets in the program it is given, the library's own text.length is not walked, and what reaches the caller is a string rather than a number. Pad ASCII text, or count the width with codePointLength yourself, when the answer must be the same on both.

A fill that is not ASCII panics, for the reason indexOfAny's set must be ASCII: natively it would be cut in the middle of a UTF-8 sequence.

const padStart: (
  text: string,
  width: i32,
  fill: string,
) => string
textstringrequired

The text to pad.

widthi32required

The length the answer is padded to, in text.length's units.

fillstringrequired

The ASCII text repeated as the padding; " " and "0" are the usual ones.

returnsstring

text with the padding in front; text unchanged when no padding is needed.

replaceAll

text with every occurrence of needle replaced by replacement.

The parts are collected and joined once rather than concatenated in the loop: s = s + t inside a loop is quadratic in time and in arena, which is the one string mistake this project has measured the cost of — 180 MB of peak memory for 88 KB of output (docs/wp14-selfhost.md §3).

An empty needle answers text unchanged, because every other answer is a choice about how many empty matches a string contains.

const replaceAll: (
  text: string,
  needle: string,
  replacement: string,
) => string
textstringrequired

The text whose occurrences are replaced.

needlestringrequired

The text looked for. Occurrences are found left to right and do not overlap: after a match the search goes on past its end.

replacementstringrequired

The text each occurrence becomes; it is not searched.

returnsstring

The text with every occurrence replaced.

split

The pieces of text between the occurrences of sep: JavaScript's text.split(sep) for a separator that is not empty. A separator at the start, at the end or next to another gives an empty piece, so split(",a,,b,", ",") is ["", "a", "", "b", ""]; split("", ",") is [""]; and a separator that never occurs gives [text]. Compare splitWhitespace, which drops the empty pieces.

The pieces are the same natively and under Node for any text and any separator, multi-byte ones included: a match is a whole UTF-8 sequence natively and a whole run of UTF-16 units under Node, and a well-formed string holds a code point's encoding only where that code point is. So there is nothing for NL8001 (--warn-portability) to warn about, and it does not; the pieces' own .lengths are still counted in each platform's units, and NL8001 reads those where the program does.

An empty separator panics. JavaScript splits a string on "" into its UTF-16 units, which natively would be its bytes, and the two answers differ on any text that is not ASCII. A separator is almost always a literal the caller chose, so an empty one is a mistake in the program, like an index out of range, and not a condition a caller could recover from.

const split: (text: string, sep: string) => string[]
textstringrequired

The text to split.

sepstringrequired

The separator, compared byte for byte; it must not be empty. Occurrences are found left to right and do not overlap.

returnsstring[]

A fresh array of the pieces, in order; never empty.

splitLimit

The first limit pieces of text between the occurrences of sep: JavaScript's text.split(sep, limit). The pieces past the limit are dropped, not joined into the last one, so splitLimit("a,b,c", ",", 2) is ["a", "b"], and a limit of 0 gives [].

A negative limit keeps every piece, as it does under Node: JavaScript reads the limit with ToUint32, so -1 is 4294967295, and any negative i32 is at least 2147483648 there, more pieces than a string can split into. So splitLimit(text, sep, -1) is split(text, sep), and the two readings agree on every i32. Refusing a negative limit would have been a panic where JavaScript answers.

An empty separator panics, whatever the limit, for the reason split gives; JavaScript would answer [] for a limit of 0, and its UTF-16 units otherwise.

const splitLimit: (
  text: string,
  sep: string,
  limit: i32,
) => string[]
textstringrequired

The text to split.

sepstringrequired

The separator, compared byte for byte; it must not be empty. Occurrences are found left to right and do not overlap.

limiti32required

The most pieces kept; a negative one keeps them all.

returnsstring[]

A fresh array of at most limit pieces, in order.

splitLines

The lines of text, without their terminators.

A trailing newline does not produce a final empty line, because a text file that ends in one has as many lines as it has newlines and a caller comparing two files line by line would otherwise see a phantom difference at the end. A \r\n line ending keeps its \r: stripping it would make this function guess at the file's provenance, and a caller that needs it gone can trimEnd each line.

const splitLines: (text: string) => string[]
textstringrequired

The text to split; an empty one has no lines.

returnsstring[]

A fresh array of the lines, in order.

splitWhitespace

The runs of non-blank bytes in text, with every blank run as the separator. Leading, trailing and repeated blanks produce no empty elements, which is what splitting a line of command-line flags wants — and it is why this is not split(text, " "), which would.

const splitWhitespace: (text: string) => string[]
textstringrequired

The text to split; the blanks are space, tab, newline and carriage return.

returnsstring[]

A fresh array of the non-blank runs, in order; empty when text is all blank.

toFixed

x written with exactly digits digits after the point: JavaScript's x.toFixed(digits), and the same text natively and under Node.

The answer is the decimal closest to the exact binary value of x, and a tie between two takes the larger, as ECMA-262 says. That is not rounding the shortest printed decimal: 1.005 is stored as a little less than 1.005, so toFixed(1.005, 2) is "1.00", while 2.5 is exact and a tie, so toFixed(2.5, 0) is "3" and toFixed(-2.5, 0) is "-3". The digits are worked out in full, one decimal digit per array element, so the value is never rounded on the way; that is why this is written here and not with a multiplication by a power of ten, which rounds twice.

The edges are JavaScript's. NaN gives "NaN". x at or beyond 1e21 in size gives String(x), the text a template hole gives, so toFixed(1e21, 2) is "1e+21", and the infinities give "Infinity" and "-Infinity". -0 loses its sign, "0.00", but a negative value that rounds to zero keeps it: toFixed(-0.0001, 2) is "-0.00".

digits outside 0 to 100 panics, where JavaScript throws a RangeError: the count is almost always a literal the caller chose, so a bad one is a mistake in the program, as split's empty separator is.

const toFixed: (x: f64, digits: i32) => string
xf64required

The number to write.

digitsi32required

How many digits to write after the point, from 0 to 100; with 0 there is no point.

returnsstring

x in fixed notation, or String(x) when x is NaN or at least 1e21 in size.

trim

text without blank bytes at either end.

const trim: (text: string) => string
textstringrequired

The text to trim; the blanks are space, tab, newline and carriage return.

returnsstring

text from its first non-blank byte to its last, or "" when it is all blank.

trimEnd

text without its trailing blank bytes.

const trimEnd: (text: string) => string
textstringrequired

The text to trim; the blanks are space, tab, newline and carriage return.

returnsstring

text up to and including its last non-blank byte, or "" when it is all blank.

trimStart

text without its leading blank bytes.

const trimStart: (text: string) => string
textstringrequired

The text to trim; the blanks are space, tab, newline and carriage return.

returnsstring

text from its first non-blank byte on, or "" when it is all blank.