nish/net/url

The request target's path and query, split and percent-decoded in place, and its query string read as key/value pairs; and whole URLs, parsed into their parts and references resolved against them.

import {
  URL_DECODE_ALLOW_NUL,
  URL_DECODE_ERR_ESCAPE,
  URL_DECODE_ERR_NUL,
  URL_DECODE_ERR_SPACE,
  URL_DECODE_FORM,
  URL_DECODE_LENIENT,
  URL_DEFAULT_MAX_PAIRS,
  URL_QUERY_END,
  URL_QUERY_ERR_PAIRS,
  URL_QUERY_PAIR,
  Url,
  UrlPathSegments,
  UrlQuery,
  UrlQueryLimits,
  percentDecode,
  percentDecodeErrorAt,
  percentDecodePathSegment,
  urlDecode,
  urlDefaultPort,
  urlIsHostPort,
  urlOrigin,
  urlParse,
  urlQueryFind,
  urlQueryGet,
  urlQueryStart,
  urlResolve,
  urlSerialize,
} from "nish/net/url";
import { UrlQuery, UrlQueryLimits, URL_DECODE_FORM, URL_QUERY_PAIR, percentDecode, urlQueryFind, urlQueryStart } from "nish/net/url";

const q: i32 = urlQueryStart(target, 0, len)    // -1: no `?`, so no query
if (q >= 0) {
  const query: UrlQuery = new UrlQuery(new UrlQueryLimits())
  query.reset(target, q + 1, len - q - 1)
  if (urlQueryFind(query, "name") === URL_QUERY_PAIR) {
    const n: i32 = percentDecode(target, query.valueStart, query.valueLength, out, 0, URL_DECODE_FORM)
    if (n >= 0) {
      // out[0, n) is the value
    } // else n is a URL_DECODE_ERR_* code: a peer's `%zz`, never read out
  }   // else no such key, or URL_QUERY_ERR_PAIRS past the cap
}

A handler that routes on a path or reads ?name=… from a query otherwise writes its own byte scan, and each such scan decides differently what %zz, %00, + or a hundred thousand & mean. This module decides once:

The byte API allocates nothing

urlQueryStart, percentDecode, UrlPathSegments and UrlQuery read a window bytes[off, off + len) and answer offsets into it, or write into the caller's array; reset reuses an iterator, so a server that holds one per connection allocates nothing per request. No byte outside the window is read. A window outside its array, or an at outside out, is a bug in the caller and panics; nothing a peer sends does.

Decoding is strict unless asked otherwise

With no flags, percentDecode refuses a % not followed by two hex digits (URL_DECODE_ERR_ESCAPE) and %00 (URL_DECODE_ERR_NUL), and copies + as it stands. URL_DECODE_FORM reads + as a space, as application/x-www-form-urlencoded does; URL_DECODE_LENIENT copies a malformed escape as it stands, as the WHATWG URL Standard's percent-decode and URLSearchParams do; URL_DECODE_ALLOW_NUL decodes %00 to a 0 byte. For valid UTF-8, URLSearchParams is URL_DECODE_FORM | URL_DECODE_LENIENT | URL_DECODE_ALLOW_NUL; it replaces an invalid sequence with U+FFFD.

Decoded bytes are bytes

The byte API does not check that what it decodes is UTF-8, where decodeURIComponent throws on %FF: a path segment or a value may be binary, and whatever reads it as text checks it. urlDecode and urlQueryGet, which answer a string, refuse anything that is not valid UTF-8.

A path is split before it is decoded

UrlPathSegments splits on the raw / only, and percentDecodePathSegment decodes one segment, so %2F is a 0x2F byte inside its segment and never starts a new one: /a%2Fb/c is the two segments a/b and c. + is literal in a path.

A query yields at most `UrlQueryLimits.maxPairs` pairs

A query yields at most UrlQueryLimits.maxPairs pairs (100 by default), and past it next() answers URL_QUERY_ERR_PAIRS and stays spent. Empty pairs (a&&b, a leading or trailing &) are skipped and not counted, as URLSearchParams skips them.

A fragment is not part of a request target

A fragment is not part of a request target (RFC 9112 §3.2), so the byte API never looks for a #: in a target that holds one, it is a byte of the path or of the last value.

A whole URL is parsed strictly, and nothing in it is decoded

A client that is handed http://… by a user, a redirect or a link needs its host and port before it connects, and the place each reference points to:

import { Url, urlOrigin, urlParse, urlResolve, urlSerialize } from "nish/net/url";

const base: Url | null = urlParse("https://example.com/a/b?q")
if (base !== null) {
  const next: Url | null = urlResolve(base, "../c#top")  // https://example.com/c#top
  // next.host, next.port (-1: the default), urlOrigin(next) === "https://example.com:443"
}

urlParse and urlResolve follow RFC 3986 (§3 for the parts, §5.2 to resolve, dot segments removed) and refuse, with null, what two readers could take for different hosts: a byte outside printable ASCII, a backslash, a malformed %, a second @, a percent-encoded or otherwise non-unreserved host byte, a host that looks numeric but is not a strict dotted-decimal IPv4 address (127.1, 0x7f.0.0.1, 2130706433), an IPv6 zone or IPvFuture literal, and a port past 65535. The scheme and host are lower-cased; every other part is kept as written, still percent-encoded.

The record of what this module refuses and why is docs/security/url.md.

Functions

percentDecode

Percent-decodes src[off, off + len) into out from out[at] on, and answers the number of bytes written, or a negative URL_DECODE_ERR_* code; flags is a sum of URL_DECODE_* (0 is strict: malformed escapes and %00 refused, + kept). Decoding never makes the input longer, so len bytes of room always suffice; with less, URL_DECODE_ERR_SPACE is answered before any write at or past out.length. On an error the bytes before it have been written. What is decoded is not checked as UTF-8.

const percentDecode: (
  src: u8[],
  off: i32,
  len: i32,
  out: u8[],
  at: i32,
  flags: i32,
) => i32
srcu8[]required

The array holding the encoded text.

offi32required

Where the encoded text starts in src.

leni32required

The encoded text's length in bytes; a window outside src panics.

outu8[]required

The array the decoded bytes are written into.

ati32required

The index in out the first decoded byte goes to; one below 0 or above out.length panics.

flagsi32required

URL_DECODE_FORM, URL_DECODE_LENIENT and URL_DECODE_ALLOW_NUL, summed, or 0 for strict.

returnsi32

How many bytes were written from out[at] on, or URL_DECODE_ERR_ESCAPE, URL_DECODE_ERR_NUL or URL_DECODE_ERR_SPACE.

percentDecodeErrorAt

The offset in src of the byte percentDecode with the same arguments fails at — the % of a malformed escape or of %00, or the first byte out has no room for — or -1 when it does not fail. It writes what percentDecode would have written before the error.

const percentDecodeErrorAt: (
  src: u8[],
  off: i32,
  len: i32,
  out: u8[],
  at: i32,
  flags: i32,
) => i32
srcu8[]required

The array holding the encoded text.

offi32required

Where the encoded text starts in src.

leni32required

The encoded text's length in bytes; a window outside src panics.

outu8[]required

The array the decoded bytes are written into, as percentDecode writes them.

ati32required

The index in out the first decoded byte goes to; one below 0 or above out.length panics.

flagsi32required

The URL_DECODE_* sum percentDecode was given.

returnsi32

The index in src decoding stops at, or -1 when all of it decodes.

percentDecodePathSegment

percentDecode for one path segment: + is literal whatever flags say, and %2F is a 0x2F byte in the segment. Split the path with UrlPathSegments first, then decode each segment.

const percentDecodePathSegment: (
  src: u8[],
  off: i32,
  len: i32,
  out: u8[],
  at: i32,
  flags: i32,
) => i32
srcu8[]required

The array holding the path.

offi32required

Where the segment starts in src, as UrlPathSegments gives it.

leni32required

The segment's length in bytes; a window outside src panics.

outu8[]required

The array the decoded bytes are written into.

ati32required

The index in out the first decoded byte goes to; one below 0 or above out.length panics.

flagsi32required

A URL_DECODE_* sum; URL_DECODE_FORM in it is ignored.

returnsi32

How many bytes were written from out[at] on, or a negative URL_DECODE_ERR_* code, as percentDecode answers.

urlDecode

s percent-decoded with flags, or null when percentDecode refuses it or what it decodes to is not valid UTF-8, which is where decodeURIComponent throws. s is expected to be ASCII, as a URL on the wire is. It allocates its answer.

const urlDecode: (s: string, flags: i32) => string | null
sstringrequired

The encoded text; a code unit past 0xFF (possible only under Node) answers null.

flagsi32required

URL_DECODE_FORM, URL_DECODE_LENIENT and URL_DECODE_ALLOW_NUL, summed, or 0 for strict.

returnsstring | null

The decoded text, or null for a refused escape or %00, or text that is not UTF-8.

urlDefaultPort

The port a scheme uses when a URL names none.

const urlDefaultPort: (scheme: string) => i32
schemestringrequired

The scheme, lower-cased and without its :, as Url.scheme holds it.

returnsi32

80 for http and ws, 443 for https and wss, and -1 for any other scheme.

urlIsHostPort

Whether bytes[a, b) is uri-host [ ":" port ] (RFC 3986 §3.2.2, §3.2.3) by the rules urlParse holds a URL's host and port to: a bracketed IPv6 address, a dotted-decimal IPv4 address, or a reg-name of unreserved bytes that does not read as a number, then an optional : and a port of digits up to 65535. An empty host and an empty port are allowed, as in a URL; a percent-encoded or sub-delims byte in a reg-name is not. It is the rule an HTTP server holds a Host field to (RFC 9112 §3.2). Allocates nothing.

const urlIsHostPort: (
  bytes: u8[],
  a: i32,
  b: i32,
) => boolean
bytesu8[]required

The bytes that hold the value.

ai32required

The index of the value's first byte.

bi32required

The index past the value's last byte.

returnsboolean

True when the bytes are a host and an optional port; false otherwise, and for a window outside bytes.

urlOrigin

u's origin as scheme://host:port, the port always written (the scheme's default when u names none), so origins compare as plain strings: http://A.com/x and http://a.com:80/y are both http://a.com:80. Null for a scheme other than http, https, ws or wss, which has no origin to compare. Hosts are compared as written, lower-cased, so [::1] and [0::1] are different origins, and so are a.com. and a.com: equal strings are the same origin, and a split errs toward "different".

const urlOrigin: (u: Url) => string | null
uUrlrequired

The URL whose origin is wanted.

returnsstring | null

The origin, such as https://example.com:443, or null for a scheme with no default port.

urlParse

The absolute URL s split into its parts, or null when it is not one this module accepts.

s is refused when it has no scheme, holds a byte outside printable ASCII (a control byte, a space, DEL or anything past 0x7F), a backslash or a % not followed by two hex digits, or its authority has more than one @, a host with a byte outside RFC 3986's unreserved set (so a percent-encoded one), a host whose last label is numeric but that is not a dotted-decimal IPv4 address, a malformed IPv6 literal or a port that is not digits or is above 65535. An http, https, ws or wss URL needs a non-empty host.

The path's dot segments are removed, as RFC 3986 §5.2.2 removes them from a reference with a scheme. Nothing is decoded. It allocates its answer.

const urlParse: (s: string) => Url | null
sstringrequired

The absolute URL, such as https://example.com/a?b#c.

returnsUrl | null

The URL's parts, still percent-encoded, scheme and host lower-cased; or null when s is refused.

urlQueryFind

Moves q on to the first of its remaining pairs whose key, decoded in form mode, equals name, and answers URL_QUERY_PAIR with q's windows on it; or URL_QUERY_END, or the error next() answered. Nothing is allocated. ?na%6De=x holds the key name; a key with a malformed escape or %00 matches nothing. Call it again for the next pair with the same key. name is ASCII: natively it is compared with the key's bytes, under Node with its UTF-16 code units, so a non-ASCII name answers differently.

const urlQueryFind: (q: UrlQuery, name: string) => i32
qUrlQueryrequired

The query iterator, moved on past the pair found; its windows bound that pair.

namestringrequired

The key to look for, as it reads decoded.

returnsi32

URL_QUERY_PAIR when a pair with the key was found, URL_QUERY_END when none is left, or the negative URL_QUERY_ERR_* code next() answered.

urlQueryGet

The first value for name in the query of the request target target, form-decoded strictly: "" for a key written without =, as URLSearchParams.get answers, and null when the target has no query, the key is absent, the value is malformed or not UTF-8, or the key comes after URL_DEFAULT_MAX_PAIRS pairs. It allocates its answer.

const urlQueryGet: (
  target: string,
  name: string,
) => string | null
targetstringrequired

A request target such as /search?q=nish; everything after its first ? is the query.

namestringrequired

The key to look for, as it reads decoded; ASCII, as urlQueryFind asks.

returnsstring | null

The first decoded value for name, "" when it has no =, or null in the cases above.

urlQueryStart

The index of the first ? in bytes[off, off + len), or -1 when there is none. The path is [off, q) and the query (q, off + len); nothing is copied. A # is not looked for: a request target holds no fragment.

const urlQueryStart: (
  bytes: u8[],
  off: i32,
  len: i32,
) => i32
bytesu8[]required

The array holding the request target.

offi32required

Where the target starts in bytes.

leni32required

The target's length in bytes; a window outside bytes panics.

returnsi32

The index in bytes of the ?, or -1 when the target has no query.

urlResolve

The reference ref resolved against the absolute URL base by RFC 3986 §5.2.2, strictly (http:g is http:g, not relative), or null when ref breaks a rule urlParse names or what it resolves to is an http, https, ws or wss URL without a host. base's fragment is never carried over. It allocates its answer and does not change base.

const urlResolve: (base: Url, ref: string) => Url | null
baseUrlrequired

The URL ref is relative to, as urlParse made it.

refstringrequired

The reference: an absolute URL, a network-path //host/p, an absolute path, a relative path, a query or a fragment.

returnsUrl | null

The resolved URL, still percent-encoded, or null when ref is refused or resolves to a URL without a host its scheme needs.

urlSerialize

u written back as RFC 3986 §5.3 recomposes it, an IPv6 host in brackets. The userinfo is written when either half is non-empty. A path that starts with // under no authority is written after /., as the WHATWG URL Standard writes it, so the string does not read back with an authority.

const urlSerialize: (u: Url) => string
uUrlrequired

The URL to write; its parts are written as they are held, so still percent-encoded.

returnsstring

The URL as one string, scheme to fragment.

Classes

Url

A URL split into its parts (RFC 3986 §3). Each part is held as it was written, still percent-encoded, except that scheme and host are lower-cased. urlParse and urlResolve make one; urlSerialize writes one back.

class Url
Member
new Url()
fragment: string | null Without its #, or null when there is no #.
hasAuthority: boolean True when the URL has a // authority; mailto:a@b has none and file:///x has an empty one.
host: string A lower-cased reg-name, a dotted IPv4 address, or an IPv6 address without its brackets.
password: string The userinfo after its first :, as written; empty when there is none.
path: string With its dot segments removed; empty, or starting with / when there is an authority.
port: i32 0 to 65535, or -1 when the URL names none; urlOrigin supplies the scheme's default.
query: string | null Without its ?, or null when there is no ?.
scheme: string Lower-cased, without its :: http.
username: string The userinfo before its first :, as written; empty when there is none.

UrlPathSegments

The segments of a path, split on the raw /. A leading / starts the path rather than an empty segment, so /a//b/ is a, , `b`, , / is one empty segment and an empty window has none. Each segment's raw bytes are bytes[start, start + length); decode one with percentDecodePathSegment.

class UrlPathSegments
Member
new UrlPathSegments() An iterator over nothing until reset.
bytes: u8[] The array the path is read from, as reset was given it.
length: i32 The current segment's length in raw bytes.
start: i32 The current segment's first byte.
next(): boolean Moves to the next segment, whose bytes start and length then name.
reset(bytes: u8[], off: i32, len: i32): void Starts over on the path bytes[off, off + len); panics when the window is outside the array.

UrlQuery

The pairs of a query string, one per next(), in order. The window is the query without its ?. & separates pairs and the first = in a pair ends its key: a&b=&c=d=e is a with no value, b with an empty one and c with d=e. A key's raw bytes are bytes[keyStart, keyStart + keyLength) and a value's bytes[valueStart, valueStart + valueLength); decode them with percentDecode and URL_DECODE_FORM.

class UrlQuery
Member
new UrlQuery(limits: UrlQueryLimits) An iterator over nothing until reset, with the cap in limits.
failed: i32 0, or the URL_QUERY_ERR_* code every later next() answers.
hasValue: boolean False for a pair written without = (a), true for a= with valueLength 0.
keyLength: i32 The current pair's key length in raw bytes.
keyStart: i32 Where the current pair's key starts in bytes.
valueLength: i32 The current pair's value length in raw bytes, 0 when it has none.
valueStart: i32 Where the current pair's value starts in bytes.
next(): i32 Moves to the next pair, setting the key and value windows.
reset(bytes: u8[], off: i32, len: i32): void Starts over on the query bytes[off, off + len), clearing failed; panics when the window is outside the array.

UrlQueryLimits

What a UrlQuery refuses to yield. The iterator copies the number when it is made; a negative cap is read as zero, so every query with a pair fails.

class UrlQueryLimits
Member
new UrlQueryLimits()
maxPairs: i32 The most pairs one query yields, URL_DEFAULT_MAX_PAIRS (a field starts at a literal); past it next() answers URL_QUERY_ERR_PAIRS and stays spent.

Constants

4

URL_DECODE_ALLOW_NUL: i32

%00 decodes to a 0 byte rather than being refused.

-1

URL_DECODE_ERR_ESCAPE: i32

A % not followed by two hex digits: %zz, or %4 or % at the end of the window.

-2

URL_DECODE_ERR_NUL: i32

%00 without URL_DECODE_ALLOW_NUL.

-3

URL_DECODE_ERR_SPACE: i32

out has no room for the next byte; nothing is written at or past out.length.

1

URL_DECODE_FORM: i32

+ is a space, as application/x-www-form-urlencoded writes one.

2

URL_DECODE_LENIENT: i32

A % not followed by two hex digits is copied as it stands rather than refused (WHATWG percent-decode).

100

URL_DEFAULT_MAX_PAIRS: i32

The default for UrlQueryLimits.maxPairs.

0

URL_QUERY_END: i32

The window holds no more pairs; every later next() answers the same.

-4

URL_QUERY_ERR_PAIRS: i32

The query holds more than maxPairs pairs; sticky in failed.

1

URL_QUERY_PAIR: i32

next() found a pair; the iterator's windows bound it.