HTTP client

Two modules speak HTTP/1.1 as a client. nish/net/fetch is the one a program wants: fetchSync takes a URL and blocks until the response is in, with a timeout on every wait and a cap on every body, and await fetch(url) does the same without blocking the process, so other tasks run while it waits (Fetching without blocking). nish/net/http1-client-codec, below it, is the protocol alone: it writes a request's head and reads the response, bytes in and events out, and opens no socket, for a program that runs its own loop. Both are imported like any other library module (export and import).

Fetching a URL

import { HttpOptions, HttpRequest, HttpResponse, fetchSync } from "nish/net/fetch";

export const main = (): i32 => {
  const req = new HttpRequest("GET", "http://example.com/");
  req.header("Accept", "text/html");
  const options = new HttpOptions();
  options.totalTimeoutMs = 20000;
  const res: HttpResponse | null = fetchSync(req, options);
  if (res === null) {
    console.error("the fetch failed");
    return 1;
  }
  const type = res.header("content-type");
  console.log(`${res.status} ${res.reason}, ${type === null ? "no type" : type}`);
  const body: u8[] | null = res.body.readAll(1048576);
  if (body === null) {
    console.error("the body failed or was larger than 1 MiB");
    return 1;
  }
  console.log(String.fromBytes(body, 0, toI32(body.length)));
  return 0;
};

fetchSync looks the host up, connects, sends the request and answers once the response's head is in: status, reason, the url that answered and header(name), in any case. The body is read through res.body: readAll(limit) answers it whole, or null past limit or on a failure, and read(buf, off, n) answers the next bytes as they arrive, 0 at the end and a negative code on a failure, for a stream you want as it comes; a read of 0 bytes answers 0 at once.

Once the body ends or fails, fetchSync closes its connection; read it to the end, or, when you only want the status, close it with res.body.client.close(), since an unread body holds its socket open until the program exits. A request carries a body when you set req.body, with its Content-Length sent for you. A header value holding a line break, or a method that is not a token, is refused before a byte is sent.

Knowing why it failed

fetchSync answers null with no reason. An FetchSyncClient does the same work, says why in error (an HTTP_E_* code; httpErrorName names it) and errorText (a sentence, such as could not connect to example.com port 80 (errno 111)), and keeps the connection alive between fetches to the same place.

A URL that errorText names is only its scheme, host, port and path, with ?... where a query or fragment was left out, and a URL that does not parse is not echoed at all. The URL is named as it parses: in http://admin:8080/s3cret@example.com/ the 8080 is a port and the rest a path, which is echoed, so put credentials in a header, not in a URL:

import { FetchSyncClient, HttpOptions, HttpRequest, HttpResponse, httpErrorName } from "nish/net/fetch";

export const main = (): i32 => {
  const client = new FetchSyncClient(new HttpOptions());
  const res: HttpResponse | null = client.fetch(new HttpRequest("GET", "http://example.com/"));
  if (res === null) {
    console.error(`${httpErrorName(client.error)}: ${client.errorText}`);
    client.close();
    return 1;
  }
  console.log(`${res.status}`);
  res.body.readAll(1048576);
  client.close();
  return 0;
};

Read each body to its end before the next fetch: the next fetch on a connection with an unread body closes it, and the old body's reader then answers an error rather than the next response's bytes. A connection is kept only when nothing is left on it past the response: a server that sends more than was asked for, or speaks while the connection sits idle, has it closed, and the next fetch opens a new one.

A kept connection the server has closed is seen before the request is sent, and the request goes on a fresh one. If the server closes it under the request, before a byte of the answer, the request is sent once more on a fresh one for a GET, HEAD, OPTIONS, TRACE, PUT or DELETE, never for a POST; a request whose answer was cut off part way, or that was never answered, is not sent again.

Fetching without blocking

In an async function, await fetch(url, init) fetches without holding up the process: while it looks the name up, connects, runs the TLS handshake of an https URL, writes and waits for the answer, the program's other tasks run (Async functions). Two fetches under awaitAll are in flight together:

import { awaitAll } from "nish:async";
import { FetchPool, RequestInit, fetch } from "nish/net/fetch";

const title = async (url: string, pool: FetchPool): Promise<string> => {
  const init = new RequestInit();
  init.header("Accept", "text/plain");
  init.pool = pool;
  const r = await fetch(url, init);
  if (r.isErr()) {
    return r.error.text;
  }
  return `${r.value.status}: ${r.value.text()}`;
};

export const main = async (): Promise<i32> => {
  const pool = new FetchPool();
  const [a, b] = await awaitAll(
    title("http://127.0.0.1:8080/a", pool),
    title("http://127.0.0.1:8080/b", pool)
  );
  console.log(a);
  console.log(b);
  pool.close();
  return 0;
};

fetch answers a Result: a FetchResponse or a FetchError, whose code is the HTTP_E_* code fetchSync's client would have set and whose text is its name and sentence, such as refused: could not connect to 127.0.0.1 port 8080 (errno 111). It never answers null and never panics. A 404 or a 500 is a response, not an error: read status.

The body is read whole before fetch answers, up to the options' maxBodyBytes (256 MiB unless you lower it; past it the fetch fails with HTTP_E_BODY_TOO_LARGE), so text() and bytes() are plain calls, not awaited. Set maxBodyBytes to what you expect from a server you do not run. A body is held in one array, so even a raised maxBodyBytes stops just under 2 GiB.

What a fetch allocates is freed when the async function that awaited it returns. A program that fetches in a long loop should await each fetch in a function of its own, as title above is, so each response goes when that function ends. A loop that awaits fetch directly in main keeps every response until main returns. header(name) reads a field in any case, url is the URL that answered.

RequestInit holds the request: method (GET unless set), fields added with header(name, value), body (a u8[], with its Content-Length sent for you), options (an HttpOptions) and pool. Pass null, or nothing, for a plain GET. The options mean what they mean to fetchSync, redirects, proxies and the outbound policy included, except for the limits on the waits:

  • Each wait on the socket is bounded by readTimeoutMs, and by what is left of totalTimeoutMs, whichever ends first: a server that never sends another byte, or never answers the TLS handshake, fails the fetch with HTTP_E_READ_TIMEOUT or HTTP_E_TOTAL_TIMEOUT.
  • A cancelled task's fetch fails with HTTP_E_CANCELLED at its next wait, whether cancel, cancelAfter or the server running the task (std/net/serve) cancelled it.
  • connectTimeoutMs bounds the name lookup, which parks only its own task: the query goes out on a non-blocking socket, with the servers, tries and timeouts of /etc/resolv.conf (or of options.dns), and the other tasks run until it answers. An address literal, localhost and a name in /etc/hosts need no query. A lookup still unanswered at connectTimeoutMs fails with HTTP_E_CONNECT_TIMEOUT.

https is the same TLS 1.3 as fetchSync's, with the same certificate checks, the same roots (rootsPemPath) and the same errors: a certificate that is expired, for another name or from a root you do not trust is HTTP_E_CERTIFICATE with the sentence fetchSync gives. Through a proxy it is a CONNECT tunnel. An https connection is never kept in a FetchPool: each https fetch runs its own handshake and closes its connection once the body is read.

A FetchPool keeps connections open between fetches: hand the same pool to each RequestInit, and a fetch to a place the pool holds an idle connection for takes it rather than dialling.

A connection is kept only when its response was read to the end and nothing is left on it, at most maxIdlePerOrigin (4 unless given to the constructor) for one origin or proxy and 64 in all. A kept connection the server closed, or wrote to unasked, is closed when it is next looked at, and a GET whose kept connection drops before a byte of the answer is sent once more on a fresh one, as fetchSync does. A fetch under an outbound policy neither keeps nor takes one. pool.idle() counts the idle connections and pool.close() closes them; without a pool, each fetch closes its own.

fetch does not run under --emit-ts: it waits on its socket with nish:async's readable and writable, which nish --emit-ts refuses with NL4017 (Async functions). A program that must also run as TypeScript uses fetchSync.

Options

HttpOptions holds every limit, set before the fetch:

Field Default What it bounds
connectTimeoutMs 10,000 The lookup and the connect, across every address the name has
readTimeoutMs 30,000 Each wait for the server to send or take a byte
totalTimeoutMs 0 (none) The whole fetch, redirects and body included
maxHeaderBytes 65,536 The status line, and the header section
maxBodyBytes 256 MiB Any body, however it is framed
followRedirects false Whether a 3xx with a Location is followed
maxRedirects 5 How many hops
crossOriginRedirects false Whether a hop may go to another origin
useEnvProxy true Whether the proxy variables are read
rootsPemPath null A PEM file of the roots an https server must chain to, in place of the system's
distrustSpkiSha256 null Keys no certificate of an https server's chain may hold, each the SHA-256 of a subjectPublicKeyInfo
outbound null The outbound policy for URLs your program did not choose
dns null The resolver settings a name is looked up with, a DnsConfig as dnsParseResolvConf fills it, in place of /etc/resolv.conf; /etc/hosts is read either way

A server that sends its body a byte at a time, each inside readTimeoutMs, holds a fetch until totalTimeoutMs: set it when the server is not yours. The addresses a name has are tried one after another inside connectTimeoutMs: an address that drops the connection attempt without an answer uses up the whole budget, so the addresses after it are not tried and the fetch fails with HTTP_E_CONNECT_TIMEOUT.

Redirects are followed only when you set followRedirects, and only within the origin unless you set crossOriginRedirects too. A hop to another origin drops Authorization, Proxy-Authorization and Cookie, but not a credential you send in a field of your own (X-Api-Key): leave crossOriginRedirects off for such a request. A hop from https to http is always refused. 303 turns the request into a GET without its body, as do 301 and 302 after a POST; 307 and 308 keep both.

Proxies come from http_proxy, https_proxy and no_proxy (or their upper-case names), unless useEnvProxy is off; client.setProxies sets them directly. HTTP_PROXY is ignored when the program runs as a CGI script, where a request's Proxy: header arrives under that name. Only an http:// proxy is used; an https request goes through it in a CONNECT tunnel. The proxy's user name and password come from its URL; a Proxy-Authorization you set yourself goes only to an http proxy, in place of the one from its URL, and never to the server.

https is TLS 1.3, and the server's certificate is always checked: its chain must end at a root you trust, it must be valid now by the wall clock, and it must name the URL's host (or its IP address, for an IP literal). The roots are read once per client, from rootsPemPath when you set it, else from the files SSL_CERT_FILE and SSL_CERT_DIR name, else from the system's bundle.

A certificate that is expired, for another name, or from a root you do not trust is HTTP_E_CERTIFICATE, and so is a fetch when no root can be read at all, refused before any connection is opened. There is no switch that turns the check off: to talk to a server with its own certificate authority, point rootsPemPath at that authority's root. A chain through a key you list in distrustSpkiSha256 is HTTP_E_CERTIFICATE too, and a hash there that is not 32 bytes fails every https fetch rather than being skipped (Distrusting a key). A server that does not speak TLS, and a body cut off without TLS's close_notify, are HTTP_E_TLS. Revocation is not checked.

Fetching a URL someone else chose

A program that fetches a URL a user typed, a webhook a customer registered, or an image a page links to is a way into its own network for whoever wrote that URL. http://169.254.169.254/ reaches a cloud host's metadata service and the credentials in it; http://127.0.0.1:6379/ reaches a database that trusts local callers; a name that resolves to either does the same. Give such a client an OutboundPolicy, from nish/net/outbound:

import { HTTP_E_BLOCKED, FetchSyncClient, HttpOptions, HttpRequest, HttpResponse } from "nish/net/fetch";
import { OutboundPolicy } from "nish/net/outbound";

export const main = (): i32 => {
  const policy = new OutboundPolicy();
  const options = new HttpOptions();
  options.outbound = policy;
  options.followRedirects = true;
  options.crossOriginRedirects = true;
  const client = new FetchSyncClient(options);
  const res: HttpResponse | null = client.fetch(new HttpRequest("GET", "http://127.0.0.1/"));
  if (res === null && client.error === HTTP_E_BLOCKED) {
    console.log(client.errorText);
  }
  client.close();
  return 0;
};

It prints blocked: 127.0.0.1 resolves to 127.0.0.1, which the outbound policy does not allow, and no connection was opened. That text names the address a name resolved to, so log it, but do not hand it back to whoever supplied the URL: it would tell them your internal addresses. Answer them with a fixed message. Under a policy:

  • Only public addresses are reached. The name is looked up first, every address that is loopback, private (10/8, 172.16/12, 192.168/16, fc00::/7), link-local (169.254/16, fe80::/10), shared (100.64/10), multicast, documentation or otherwise reserved is dropped, and the client connects only to the addresses it kept, without looking the name up again. A name whose answer changes between the check and the connect cannot move the connection. An IPv4 address written inside IPv6 (::ffff:127.0.0.1, 64:ff9b::7f00:1, 2002:7f00:1::) is judged as the IPv4 address it holds; a NAT64 prefix of your network's own choosing is not recognised. A name with no public address is HTTP_E_BLOCKED.
  • Every redirect hop is checked the same way, so a public URL that answers with a redirect to http://localhost/ fails at the hop.
  • No proxy is used. A proxy looks the name up itself, past the check, so the proxy variables and setProxies are ignored.
  • A fetch is capped at 5 redirects, 30 seconds and an 8 MiB body, or your options' limits where they are lower. Change them on the policy: maxRedirects, totalTimeoutMs (0 for your options' alone) and maxBodyBytes.

Two methods change what gets through. policy.permit("10.20.0.0/16") lets a prefix you trust through anyway, such as an internal API the program is meant to reach (its IPv4-mapped, NAT64 and 6to4 forms too); it answers false for text that is not an address and a prefix length.

policy.allowHost("api.example.com") limits the names a URL, or any hop, may carry: once one is added, a host that is not on the list is HTTP_E_BLOCKED before it is looked up. A name is matched exactly, in any case and with or without its trailing dot; an entry with a leading dot, .example.com, also allows every name under it. An allowed host must still resolve to an allowed address.

Only this client takes a policy: a connection opened another way, with tcpConnect or another protocol's client, has no address check. A name you did not choose is still looked up, so its nameserver sees the query; an allowlist is the way to refuse a name before that. Set the policy before the client's first fetch: a connection the client keeps from a fetch made under another policy is closed rather than reused, so changing it costs a new connection.

The policy decides where a fetch may go; it does not make the response safe to trust. Read the body under your own limits, as for any server you do not run.

The protocol alone

nish/net/http1-client-codec is the client side of HTTP/1.1 without the socket: it writes a request's head and reads the response, bytes in and events out. It opens no socket and knows no TLS: your program sends the bytes the writer answers, and feeds the parser whatever it receives, cut anywhere. The event constants are shared with the server side and come from nish/net/http1-codec.

A first request

import { HTTP1_BODY, HTTP1_END, HTTP1_ERROR, HTTP1_HEAD, HTTP1_NEED_MORE, HTTP1_NO_BODY } from "nish/net/http1-codec";
import { Http1ResponseParser, http1RequestHead } from "nish/net/http1-client-codec";

const bytesOf = (text: string): u8[] => {
  const out: u8[] = [];
  const n: i32 = toI32(text.length);
  for (let i: i32 = 0; i < n; i++) {
    out.push(toU8(text.charCodeAt(i)));
  }
  return out;
};

// Hands the parser every event its input holds, and answers whether the response ended.
const drain = (p: Http1ResponseParser): boolean => {
  let event: i32 = p.next();
  while (event !== HTTP1_NEED_MORE) {
    if (event === HTTP1_HEAD) {
      const type = p.header("content-type");
      console.log(`status ${p.status}, ${type === null ? "no type" : type}`);
    } else if (event === HTTP1_BODY) {
      console.log(`body: ${String.fromBytes(p.body, p.bodyOff, p.bodyOff + p.bodyLen)}`);
    } else if (event === HTTP1_END) {
      console.log(`end, keep-alive ${p.keepAlive()}`);
      return true;
    } else if (event === HTTP1_ERROR) {
      console.log(`refused with ${p.error}: ${p.errorText}`);
      return true;
    }
    event = p.next();
  }
  return false;
};

export const main = (): i32 => {
  const head = http1RequestHead("GET", "/hello", ["Host", "Accept"], ["example.com", "text/plain"], toI64(HTTP1_NO_BODY));
  if (head === null) {
    return 1;
  }
  console.log(String.fromBytes(head, 0, toI32(head.length)));

  // What a server might answer, arriving in two pieces.
  const reply = bytesOf("HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 5\r\n\r\nhello");
  const p = new Http1ResponseParser(1024, 16384, 64, toI64(1048576), 4096);
  p.expect(false, false);
  p.feed(reply, 0, 20);
  drain(p);
  p.feed(reply, 20, toI32(reply.length) - 20);
  drain(p);
  return 0;
};

http1RequestHead answers the request line and the header section, Host first and your fields after it in their order, or null for anything it refuses. The parser is made once for a connection with its limits, told what the request was with expect, and fed; next() answers one event at a time until it answers HTTP1_NEED_MORE, which asks for more input. The events and the body are the same however the response is cut: the first drain above sees nothing, since the header section is not whole yet.

The request head

The last argument says how the body is framed, and the writer adds the field for it, after yours:

bodyLength The head ends with
n, 0 or more Content-Length: n
HTTP1_CHUNKED (-1) Transfer-Encoding: chunked; write each piece with http1Chunk and the end with http1LastChunk, from nish/net/http1-codec
HTTP1_NO_BODY (-2) neither, for a request without a body, such as a GET

http1WriteRequestHead(out, at, method, target, names, values, bodyLength) writes the same head into your own u8[] at at and answers the offset just past it. It answers HTTP1_NO_ROOM when the head does not fit and HTTP1_REFUSED for what http1RequestHead answers null for; either way it writes nothing.

The writer refuses whatever would let one of your strings end a line or frame the body on its own:

  • a method that is not a token;
  • a target that is not a form the method may send: a path starting with / or an absolute URL, host:port for CONNECT, * for OPTIONS; a target holding a space, a control byte, a non-ASCII byte or a # (a host:port sent with GET is not refused: it is a well-formed absolute URL whose scheme is the host);
  • a field name that is not a token, or a value holding CR, LF or another control byte;
  • anything but exactly one Host field;
  • your own Content-Length or Transfer-Encoding, which bodyLength writes;
  • a Connection field that lists upgrade;
  • names and values of different lengths, or a bodyLength below -2.

The events

Event What it means
HTTP1_INFORMATIONAL an interim 1xx response was read; status and the fields are its, and the final response follows. It comes from nish/net/http1-client-codec
HTTP1_HEAD the final response's status line and fields are in: status, reason, minor (the HTTP minor version), names and values (names in lowercase), header(name) (any case), contentLength (the body's length, or -1 when the response is not framed by one; it is always -1 for a response to HEAD, a 204, a 304 and a 2xx to CONNECT, which have no body, so read header("content-length") there)
HTTP1_BODY some body bytes, de-chunked: body[bodyOff .. bodyOff + bodyLen), valid until the next call
HTTP1_END the response is complete, trailers included
HTTP1_UPGRADE a CONNECT was answered with 2xx, or a request expectUpgrade named with 101, and the connection is a tunnel: tunnelBytes() answers what was fed past the head
HTTP1_CLOSED the connection is done: the server closed it between responses, or the response before was not kept alive (keepAlive() false) and nothing more can follow it
HTTP1_ERROR the response is refused: error holds the code and errorText says why

HTTP1_UPGRADE, HTTP1_CLOSED and HTTP1_ERROR are final: every later call answers them again. What is fed after HTTP1_CLOSED or HTTP1_ERROR is dropped; what is fed after HTTP1_UPGRADE is kept for tunnelBytes().

How a response ends

  • A response to HEAD, an interim 1xx, a 204 and a 304 have no body, whatever their fields say. Call p.expect(true, false) before a response to HEAD.
  • A 2xx to CONNECT is a tunnel; call p.expect(false, true) first. It answers HTTP1_HEAD, then HTTP1_END with no body, then HTTP1_UPGRADE on the next call; tunnelBytes() means something only after HTTP1_UPGRADE, and answers what was fed past the head. Any other answer to CONNECT has a body like any response.
  • Transfer-Encoding: chunked is de-chunked; Content-Length is that many bytes.
  • Anything else is read until the server closes. Call p.finish() when the socket reports the close, then next() again.

A close anywhere else, inside a status line, the fields or a body, is the error HTTP1_E_TRUNCATED, so a body cut short is never taken for a whole one. expect holds for one response and is forgotten after its HTTP1_END, so a program sending a second request on the connection calls it again.

keepAlive() answers whether the connection may carry the next request once HTTP1_END has come: an HTTP/1.1 response unless Connection lists close, an HTTP/1.0 response only when it lists keep-alive, and never a response read until close, a tunnel or an error. On a kept-alive connection, keep feeding the same parser; the next response follows the last.

What the parser refuses

A response the parser cannot read in exactly one way is refused, never repaired, since a client that repairs it differently from a proxy on the path disagrees with it about where the next response starts. Each code is a constant of nish/net/http1-client-codec:

error Code Refused
1 HTTP1_E_STATUS_LINE a malformed status line, or a status outside 100 to 599
2 HTTP1_E_VERSION an HTTP major version other than 1
3 HTTP1_E_BARE_LF a line ended by LF without CR
4 HTTP1_E_FIELD a malformed field line: a folded line, whitespace before the colon, a control byte in a value
5 HTTP1_E_SWITCH a 101 to a request that did not ask to switch protocols (expectUpgrade, below)
6 HTTP1_E_FRAMING Transfer-Encoding together with Content-Length
7 HTTP1_E_CODING a transfer coding other than one chunked, last, or one in an HTTP/1.0 response
8 HTTP1_E_LENGTH a Content-Length that is not one run of digits, or two that differ
9 HTTP1_E_CHUNK a malformed chunk size, extension or chunk ending
10 HTTP1_E_TRUNCATED the connection closed before the response ended
11 HTTP1_E_STATUS_LINE_LIMIT a status line over maxStatusLine
12 HTTP1_E_HEADER_LIMIT a header section over maxHeaderBytes
13 HTTP1_E_COUNT_LIMIT more fields than maxHeaders, trailers included
14 HTTP1_E_BODY_LIMIT a body over maxBody, refused before a byte of it is read when its length is declared
15 HTTP1_E_TRAILER_LIMIT a trailer section over maxTrailerBytes
16 HTTP1_E_CHUNK_LINE_LIMIT a chunk-size line over HTTP1_MAX_CHUNK_LINE (4,096 bytes)
17 HTTP1_E_INFORMATIONAL_LIMIT more than HTTP1_MAX_INFORMATIONAL (32) interim responses to one request

After an error the connection cannot be trusted: close it.

Switching protocols

A request that asks the server to switch protocols, such as a WebSocket's opening handshake, is written with http1UpgradeRequestHead(method, target, names, values, upgrade), which adds Upgrade and Connection: Upgrade itself. Tell the parser before you feed it the answer: call expectUpgrade() after expect. A 101 then answers HTTP1_HEAD, HTTP1_END and HTTP1_UPGRADE, and tunnelBytes() answers whatever was fed past its head, which is the new protocol's, not HTTP's. Any other answer reads as it would without the call. It lasts one request: the next expect clears it, and it does nothing once a response has begun. WebSocket shows the whole handshake.

Limits

new Http1ResponseParser(maxStatusLine, maxHeaderBytes, maxHeaders, maxBody, maxTrailerBytes) takes every limit a server could push against. maxBody is an i64; the rest are i32. A negative limit is taken as 0. What the parser holds is what you fed and it has not handed out yet, which buffered() answers: feed what the socket gave you, then drain next(), and the parser never holds more than a line in progress.

Each response's reason phrase and fields are kept as strings in the arena until your program resets it, at most maxStatusLine + maxHeaderBytes for each response.

Where next