nish/text
The string operations a program has to write itself.
import {
asciiLower,
asciiUpper,
codePointLength,
contains,
firstDifference,
indexOfAny,
padEnd,
padStart,
replaceAll,
split,
splitLimit,
splitLines,
splitWhitespace,
toFixed,
trim,
trimEnd,
trimStart,
} from "nish/text";
import { asciiUpper, padStart, replaceAll, split, splitLines, splitWhitespace, trim } from "nish/text";
export const main = (): i32 => {
const lines: string[] = splitLines("a b\n c \n"); // ["a b", " c "]
const words: string[] = splitWhitespace(lines[0]); // ["a", "b"]
const c: string = trim(lines[1]); // "c"
const dashed: string = replaceAll("a b", " ", "-"); // "a-b"
const fields: string[] = split("a,,b", ","); // ["a", "", "b"]
const loud: string = asciiUpper("get_id"); // "GET_ID"
const code: string = padStart("7", 3, "0"); // "007"
return toI32(words.length);
};
asciiLower
text with its ASCII letters A to Z made a to z, and nothing else changed: every other byte, a UTF-8 sequence's included, passes through.
asciiUpper
text with its ASCII letters a to z made A to Z, and nothing else changed: every other byte, a UTF-8 sequence's included, passes through.
codePointLength
The number of Unicode code points in text: what JSON Schema's minLength and maxLength count, and not what text.length counts.
contains
Whether needle occurs anywhere in haystack.
firstDifference
The index of the first line at which left and right differ, or -1 when one is a prefix of the other and they have the same length — that is, when they are equal.
indexOfAny
The first index at or after from whose byte is one of bytes, or -1.
padEnd
text with fill repeated after it until it is width units long: JavaScript's text.padEnd(width, fill) for ASCII text, with padStart's rules.
padStart
text with fill repeated in front of it until it is width units long: JavaScript's text.padStart(width, fill) for ASCII text.
replaceAll
text with every occurrence of needle replaced by replacement.
split
The pieces of text between the occurrences of sep: JavaScript's text.split(sep) for a separator that is not empty.
splitLimit
The first limit pieces of text between the occurrences of sep: JavaScript's text.split(sep, limit).
splitLines
The lines of text, without their terminators.
splitWhitespace
The runs of non-blank bytes in text, with every blank run as the separator.
toFixed
x written with exactly digits digits after the point: JavaScript's x.toFixed(digits), and the same text natively and under Node.
trim
text without blank bytes at either end.
trimEnd
text without its trailing blank bytes.
trimStart
text without its leading blank bytes.
The language gives a string length, charCodeAt, substring, indexOf,
startsWith, endsWith and === by content, and an array join. It does
not give split, trim, toLowerCase or a regular expression (that is
std/regex), and that is a decision rather than a gap: each allocates, and
several need a locale or a character table this runtime has no room for
(docs/wp7-runtime.md keeps the runtime under a measured byte budget).
So a program that reads a file and wants its lines writes the loop
This
module is that loop, written once. Every offset here is a byte offset,
like s.length itself, and every function but one is ASCII-only where it
inspects characters at all — a UTF-8 string passes through splitLines,
splitWhitespace, asciiUpper and asciiLower untouched, because an ASCII
byte cannot appear inside a multi-byte sequence. The one is
codePointLength, which counts code points rather than bytes and answers the
same natively and under Node; it says how. padStart and padEnd count a
width in s.length's units, so they are the two whose answers differ under
Node on text that is not ASCII; each says by how much.
Every width this module declares is spelled
Every width this module declares is spelled (i32, never number) and
every value a builtin hands it is converted at the read: s.length,
a.length, s.charCodeAt(i) and s.indexOf(t) all answer number
(LANGUAGE.md, Arrays and strings as receivers), and number is f64 under
--number-mode f64. Spelling the declarations is therefore necessary and not
sufficient — without the toI32 at each read this module does not compile in
that mode at all (docs/wp26-stdlib.md §4). A length a loop tests on every
iteration is read into an i32 local once rather than converted per
iteration, which is both cheaper and how the loop wants to read; a string is
immutable, so its length cannot change underneath the local.
Functions
asciiLower
text with its ASCII letters A to Z made a to z, and nothing else
changed: every other byte, a UTF-8 sequence's included, passes through.
This is not JavaScript's toLowerCase, for the reason asciiUpper gives:
"É" stays "É" here. On ASCII text the two agree, and this answers the
same natively and under Node for any text.
const asciiLower: (text: string) => string
textstringrequiredThe text whose ASCII letters are lower-cased.
returnsstringA fresh string as long as text.
asciiUpper
text with its ASCII letters a to z made A to Z, and nothing else
changed: every other byte, a UTF-8 sequence's included, passes through.
This is not JavaScript's toUpperCase, which knows Unicode's case table:
that gives "SS" for "ß" and "É" for "é", where this gives "ß" and
"é". That table is more than this runtime has room for, and the text this
is for, identifiers, keywords and protocol tokens, is ASCII. On ASCII text
the two agree, and this answers the same natively and under Node for any
text, since a byte or unit from 97 to 122 is always the letter itself, so
NL8001 (--warn-portability) has nothing to warn about, and does not.
const asciiUpper: (text: string) => string
textstringrequiredThe text whose ASCII letters are upper-cased.
returnsstringA fresh string as long as text.
codePointLength
The number of Unicode code points in text: what JSON Schema's
minLength and maxLength count, and not what text.length counts.
codePointLength("é") is 1 where "é".length is 2, and
codePointLength("😀") is 1 where its length is 4.
Natively a string is its UTF-8 bytes, and every code point has exactly one
byte that is not a continuation byte (10xxxxxx), so that is what is
counted. Under Node the same charCodeAt reads UTF-16 units, where a byte
rule would miscount: "©®±" is three units in 0x80–0xBF, which are
continuation bytes in UTF-8 but whole characters here. So the function asks
which host it is on (textUnitsAreBytes) and, under Node, counts every unit
except the low half of a surrogate pair.
Both walks answer the same for any string literal, a lone surrogate escape
included: it is three WTF-8 bytes with one leading byte natively, and one
unit under Node. They differ for text built from bytes: String.fromBytes
and String.fromCharCode give one unit per byte under Node, so a string
decoded that way, as nish/json decodes a \u escape, counts its UTF-8
bytes there. They also differ for text that is not well-formed — bytes that
are not UTF-8 natively, where each stray continuation byte counts nothing, or
the two halves of a pair joined at run time, which are one string under Node
and two lone surrogates natively.
const codePointLength: (text: string) => i32
textstringrequiredThe text whose code points are counted.
returnsi32How many code points text holds, counted as above.
contains
Whether needle occurs anywhere in haystack. One indexOf, named, because
indexOf(...) >= 0 at a call site reads as arithmetic where the question is a
yes or a no. contains(s, "") is true, as indexOf("") is 0.
const contains: (
haystack: string,
needle: string,
) => boolean
haystackstringrequiredThe text searched.
needlestringrequiredThe text looked for, compared byte for byte.
returnsbooleantrue when needle occurs in haystack.
firstDifference
The index of the first line at which left and right differ, or -1 when
one is a prefix of the other and they have the same length — that is, when
they are equal. A caller comparing generated text against a golden wants the
first differing line and not a boolean, because printing two whole files is
not a diagnostic.
const firstDifference: (
left: string[],
right: string[],
) => i32
leftstring[]requiredOne side's lines.
rightstring[]required
The other side's lines, compared to left's with !== by
content.
returnsi32
The index of the first line that differs, the shorter length when
one side is a prefix of the other, or -1 when they are equal.
indexOfAny
The first index at or after from whose byte is one of bytes, or -1.
bytes is a set of 1 to 16 ASCII bytes, and anything else panics: an empty
set matches nothing, more than 16 is more than one vector of the runtime
kernel compares, and an ASCII set is what keeps the answer the same under Node,
where charCodeAt reads a UTF-16 unit rather than a byte. An ASCII byte
never occurs inside a multi-byte UTF-8 sequence, so a non-ASCII text
passes through and only its ASCII bytes can match — as in splitLines, the
index is a byte offset natively. from is clamped to [0, text.length],
as substring clamps, so a negative from searches from the start and one
past the end answers -1.
Natively the search is the runtime's nish_str_index_of_any, which reads
16 or 32 bytes at a time (docs/LANGUAGE.md, "Byte search"). It is
variable-time: how long it takes says where the first match is, so it is
never for secret data.
const indexOfAny: (
text: string,
bytes: string,
from: i32,
) => i32
textstringrequiredThe text searched.
bytesstringrequiredThe set of bytes looked for, written as a string of 1 to 16 ASCII characters; their order does not matter.
fromi32required
Where the search starts, as a byte offset, clamped into
[0, text.length].
returnsi32
The byte offset of the first match at or after from, or -1 when
there is none.
padEnd
text with fill repeated after it until it is width units long:
JavaScript's text.padEnd(width, fill) for ASCII text, with padStart's
rules. padEnd("7", 4, "ab") is "7aba", and padEnd("é", 3, " ") is
"é " natively and "é " under Node.
const padEnd: (
text: string,
width: i32,
fill: string,
) => string
textstringrequiredThe text to pad.
widthi32requiredThe length the answer is padded to, in text.length's units.
fillstringrequiredThe ASCII text repeated as the padding.
returnsstring
text with the padding after it; text unchanged when no
padding is needed.
padStart
text with fill repeated in front of it until it is width units long:
JavaScript's text.padStart(width, fill) for ASCII text. A width no
longer than text, or an empty fill, gives text unchanged, and a fill of
several characters repeats and is cut at the width, so
padStart("7", 4, "ab") is "aba7".
width counts the units text.length counts: bytes natively and UTF-16
units under Node. So on text that is not ASCII the two answers differ:
padStart("é", 3, " ") is " é" natively, where é is two bytes, and
" é" under Node, where it is one unit. NL8001 (--warn-portability)
does not warn about it: the warning reads offsets in the program it is given,
the library's own text.length is not walked, and what reaches the caller is
a string rather than a number. Pad ASCII text, or count the width with
codePointLength yourself, when the answer must be the same on both.
A fill that is not ASCII panics, for the reason indexOfAny's set must
be ASCII: natively it would be cut in the middle of a UTF-8 sequence.
const padStart: (
text: string,
width: i32,
fill: string,
) => string
textstringrequiredThe text to pad.
widthi32requiredThe length the answer is padded to, in text.length's units.
fillstringrequired
The ASCII text repeated as the padding; " " and "0" are
the usual ones.
returnsstring
text with the padding in front; text unchanged when no
padding is needed.
replaceAll
text with every occurrence of needle replaced by replacement.
The parts are collected and joined once rather than concatenated in the loop:
s = s + t inside a loop is quadratic in time and in arena, which is the one
string mistake this project has measured the cost of — 180 MB of peak memory
for 88 KB of output (docs/wp14-selfhost.md §3).
An empty needle answers text unchanged, because every other answer is a
choice about how many empty matches a string contains.
const replaceAll: (
text: string,
needle: string,
replacement: string,
) => string
textstringrequiredThe text whose occurrences are replaced.
needlestringrequiredThe text looked for. Occurrences are found left to right and do not overlap: after a match the search goes on past its end.
replacementstringrequiredThe text each occurrence becomes; it is not searched.
returnsstringThe text with every occurrence replaced.
split
The pieces of text between the occurrences of sep: JavaScript's
text.split(sep) for a separator that is not empty. A separator at the
start, at the end or next to another gives an empty piece, so
split(",a,,b,", ",") is ["", "a", "", "b", ""]; split("", ",") is
[""]; and a separator that never occurs gives [text]. Compare
splitWhitespace, which drops the empty pieces.
The pieces are the same natively and under Node for any text and any
separator, multi-byte ones included: a match is a whole UTF-8 sequence
natively and a whole run of UTF-16 units under Node, and a well-formed
string holds a code point's encoding only where that code point is. So
there is nothing for NL8001 (--warn-portability) to warn about, and it
does not; the pieces' own .lengths are still counted in each platform's
units, and NL8001 reads those where the program does.
An empty separator panics. JavaScript splits a string on "" into its
UTF-16 units, which natively would be its bytes, and the two answers differ
on any text that is not ASCII. A separator is almost always a literal the
caller chose, so an empty one is a mistake in the program, like an index out
of range, and not a condition a caller could recover from.
const split: (text: string, sep: string) => string[]
textstringrequiredThe text to split.
sepstringrequiredThe separator, compared byte for byte; it must not be empty. Occurrences are found left to right and do not overlap.
returnsstring[]A fresh array of the pieces, in order; never empty.
splitLimit
The first limit pieces of text between the occurrences of sep:
JavaScript's text.split(sep, limit). The pieces past the limit are
dropped, not joined into the last one, so splitLimit("a,b,c", ",", 2) is ["a", "b"], and a
limit of 0 gives [].
A negative limit keeps every piece, as it does under Node: JavaScript
reads the limit with ToUint32, so -1 is 4294967295, and any negative
i32 is at least 2147483648 there, more pieces than a string can split
into. So splitLimit(text, sep, -1) is split(text, sep), and the two
readings agree on every i32. Refusing a negative limit would have been a
panic where JavaScript answers.
An empty separator panics, whatever the limit, for the reason split
gives; JavaScript would answer [] for a limit of 0, and its UTF-16 units
otherwise.
const splitLimit: (
text: string,
sep: string,
limit: i32,
) => string[]
textstringrequiredThe text to split.
sepstringrequiredThe separator, compared byte for byte; it must not be empty. Occurrences are found left to right and do not overlap.
limiti32requiredThe most pieces kept; a negative one keeps them all.
returnsstring[]A fresh array of at most limit pieces, in order.
splitLines
The lines of text, without their terminators.
A trailing newline does not produce a final empty line, because a text
file that ends in one has as many lines as it has newlines and a caller
comparing two files line by line would otherwise see a phantom difference at
the end. A \r\n line ending keeps its \r: stripping it would make this
function guess at the file's provenance, and a caller that needs it gone can
trimEnd each line.
const splitLines: (text: string) => string[]
textstringrequiredThe text to split; an empty one has no lines.
returnsstring[]A fresh array of the lines, in order.
splitWhitespace
The runs of non-blank bytes in text, with every blank run as the separator.
Leading, trailing and repeated blanks produce no empty elements, which is what
splitting a line of command-line flags wants — and it is why this is not
split(text, " "), which would.
const splitWhitespace: (text: string) => string[]
textstringrequiredThe text to split; the blanks are space, tab, newline and carriage return.
returnsstring[]
A fresh array of the non-blank runs, in order; empty when text is
all blank.
toFixed
x written with exactly digits digits after the point: JavaScript's
x.toFixed(digits), and the same text natively and under Node.
The answer is the decimal closest to the exact binary value of x, and
a tie between two takes the larger, as ECMA-262 says. That is not rounding
the shortest printed decimal: 1.005 is stored as a little less than
1.005, so toFixed(1.005, 2) is "1.00", while 2.5 is exact and a tie, so
toFixed(2.5, 0) is "3" and toFixed(-2.5, 0) is "-3". The digits are
worked out in full, one decimal digit per array element, so the value is
never rounded on the way; that is why this is written here and not with a
multiplication by a power of ten, which rounds twice.
The edges are JavaScript's. NaN gives "NaN". x at or beyond 1e21 in
size gives String(x), the text a template hole gives, so
toFixed(1e21, 2) is "1e+21", and the infinities give "Infinity" and
"-Infinity". -0 loses its sign, "0.00", but a negative value that
rounds to zero keeps it: toFixed(-0.0001, 2) is "-0.00".
digits outside 0 to 100 panics, where JavaScript throws a
RangeError: the count is almost always a literal the caller chose, so a
bad one is a mistake in the program, as split's empty separator is.
const toFixed: (x: f64, digits: i32) => string
xf64requiredThe number to write.
digitsi32requiredHow many digits to write after the point, from 0 to 100; with 0 there is no point.
returnsstring
x in fixed notation, or String(x) when x is NaN or at least
1e21 in size.
trim
text without blank bytes at either end.
const trim: (text: string) => string
textstringrequiredThe text to trim; the blanks are space, tab, newline and carriage return.
returnsstring
text from its first non-blank byte to its last, or "" when it is
all blank.
trimEnd
text without its trailing blank bytes.
const trimEnd: (text: string) => string
textstringrequiredThe text to trim; the blanks are space, tab, newline and carriage return.
returnsstring
text up to and including its last non-blank byte, or "" when it
is all blank.
trimStart
text without its leading blank bytes.
const trimStart: (text: string) => string
textstringrequiredThe text to trim; the blanks are space, tab, newline and carriage return.
returnsstring
text from its first non-blank byte on, or "" when it is all
blank.

