nish/net/fetch

An HTTP/1.1 client. fetchSync(request, options) is the blocking one for command-line programs: it dials, sends, and answers the response's head with a reader for its body, and a FetchSyncClient does the same while keeping one connection alive between fetches. await fetch(url, init) is the async one: it parks its task on nish:async instead of the thread, so awaitAll(fetch(a), fetch(b)) overlaps the two.

import {
  FetchError,
  FetchPool,
  FetchResponse,
  FetchSyncClient,
  HTTP_E_BLOCKED,
  HTTP_E_BODY_TOO_LARGE,
  HTTP_E_CANCELLED,
  HTTP_E_CERTIFICATE,
  HTTP_E_CONNECT_TIMEOUT,
  HTTP_E_DNS,
  HTTP_E_IO,
  HTTP_E_NONE,
  HTTP_E_PROTOCOL,
  HTTP_E_PROXY,
  HTTP_E_PROXY_REFUSED,
  HTTP_E_READ_TIMEOUT,
  HTTP_E_REDIRECT_REFUSED,
  HTTP_E_REFUSED,
  HTTP_E_REQUEST,
  HTTP_E_TLS,
  HTTP_E_TOO_MANY_REDIRECTS,
  HTTP_E_TOTAL_TIMEOUT,
  HTTP_E_URL,
  HttpBodyReader,
  HttpOptions,
  HttpRequest,
  HttpResponse,
  RequestInit,
  fetch,
  fetchSync,
  httpChooseProxy,
  httpErrorName,
  httpNoProxyMatch,
  httpProxyFor,
} from "nish/net/fetch";
import { fetch } from "nish/net/fetch";

const r = await fetch("http://127.0.0.1:8080/health");
if (r.isOk()) {
  const body: string = r.value.text();
}

The async fetch answers a Result: a FetchResponse whose body was read whole before it answered (up to maxBodyBytes, 256 MiB by default, else HTTP_E_BODY_TOO_LARGE), or a FetchError with the HTTP_E_* code. The name lookup, the connect, the TLS handshake, every write and every read wait on the executor, so a slow resolver or server stalls only its own task. An https:// URL is checked as fetchSync checks it: the caller's roots or the system's, the host name and the wall clock, with no setting that skips it. A FetchPool handed in through RequestInit.pool keeps idle http connections per route, capped; an https one is closed after its fetch, never pooled. Policy, proxies, redirects and error codes are the shared code below, so both clients behave alike. It does not run under --emit-ts.

import { HttpOptions, HttpRequest, HttpResponse, fetchSync } from "nish/net/fetch";

const req = new HttpRequest("GET", "http://example.com/");
req.header("Accept", "application/json");
const res: HttpResponse | null = fetchSync(req, new HttpOptions());
if (res !== null) {
  const body: u8[] | null = res.body.readAll(1048576);
}
It blocks the thread

It blocks the thread at every step, the way a CLI wants it to: the lookup, the connect, each write and each read. A server, which must not stall every connection for one slow peer, drives nish/net/http1-client-codec's parser from its own loop instead. Every wait is bounded: connectTimeoutMs for the lookup and the connect (one budget across every address the name has), readTimeoutMs for each wait on the peer, and totalTimeoutMs, when set, for the whole fetch, redirects and body included, so a server that drips one byte a second cannot hold a fetch forever.

Framing is the parser's

The head is written by http1RequestHead and every byte read goes through Http1ResponseParser, so a value holding a CR or LF is refused rather than sent, and every response-smuggling shape is refused with the parser's own code (see docs/security/http1-client-codec.md).

Keep-alive

Keep-alive holds one connection, to the last origin (or proxy) used. A fetch to the same place reuses it once the last body has been read to its end; a fetch while a body is unread closes it, and the old reader answers an error. A reused connection the server closed while it sat idle is the one failure a client cannot avoid, so a request with an idempotent method that fails on it before any response byte arrives is sent once more on a fresh connection (RFC 9110 §9.2.2). A POST is never sent twice.

Redirects

Redirects are not followed unless followRedirects is set, and then only to the same origin unless crossOriginRedirects is set too. A hop to another origin drops Authorization, Proxy-Authorization, Cookie and the caller's Host; a hop from https to http is always refused; 303 turns any method but HEAD into a GET without a body, as do 301 and 302 after a POST, and 307 and 308 keep the method and the body (RFC 9110 §15.4). Past maxRedirects hops the fetch fails.

Proxies

Proxies come from the environment when useEnvProxy is set: http_proxy (or HTTP_PROXY), https_proxy (or HTTPS_PROXY) and no_proxy (or NO_PROXY), the lower-case name first. HTTP_PROXY is ignored when REQUEST_METHOD is set, since a CGI program finds a request's Proxy: header there under that name (the "httpoxy" hole). Only an http:// proxy is accepted. An http request goes to it in absolute form; an https one goes through a CONNECT tunnel.

Every failure answers null with a distinct HTTP_E_* code in the client's error and a sentence in errorText; a body reader answers the code negated.

Private helpers share the importing program's flat symbol namespace (docs/wp26-stdlib.md §3e), which is why each one carries the module's name.

Written from RFC 9110 and RFC 9112, not ported from another implementation.

Functions

fetch

Fetches url without blocking the process: the task waits on the executor while the connection is made, the request written and the response read, so other tasks run meanwhile and awaitAll(fetch(a), fetch(b)) overlaps the two. The name lookup does not block either: it runs on non-blocking sockets within connectTimeoutMs. An https URL gets the TLS handshake and checks fetchSync gives it. The body is read whole, up to maxBodyBytes, before the answer comes. Policy, proxy, redirect and error rules are fetchSync's.

const fetch: (
  url: string,
  init: RequestInit | null = null,
) => Promise<Result<FetchResponse, FetchError>>
urlstringrequired

The absolute http:// or https:// URL to fetch.

initRequestInit | null

The method, fields, body, options and pool; null for a plain GET.

returnsPromise<Result<FetchResponse, FetchError>>

The response with its body, or the failure: never a panic.

fetchSync

One fetch on a client of its own, closed once the body has been read or the fetch failed. Null on a failure, with no way to say why: a program that reports its errors fetches through a FetchSyncClient.

const fetchSync: (
  req: HttpRequest,
  options: HttpOptions,
) => HttpResponse | null
reqHttpRequestrequired

The request to send, as a FetchSyncClient takes it.

optionsHttpOptionsrequired

The client's limits, timeouts, TLS trust and proxies for this one fetch.

returnsHttpResponse | null

The response, the last one when redirects were followed, or null on any failure.

httpChooseProxy

A proxy variable's value: lower when it is set and not empty, else upper when upperAllowed, else "". HTTP_PROXY is not allowed under CGI.

const httpChooseProxy: (
  lower: string | null,
  upper: string | null,
  upperAllowed: boolean,
) => string
lowerstring | nullrequired

The lower-case variable's value, such as http_proxy's, or null when it is unset.

upperstring | nullrequired

The upper-case variable's value, such as HTTP_PROXY's, or null when it is unset.

upperAllowedbooleanrequired

Whether the upper-case variable may be read: false for HTTP_PROXY under CGI, where a request's Proxy header sets it.

returnsstring

The chosen variable's value, a proxy URL or for no_proxy a comma-separated host list, or "" when neither applies.

httpErrorName

A short name for an HTTP_E_* code, for messages: "dns", "read timeout", ... A fetch's errorText starts with it.

const httpErrorName: (code: i32) => string
codei32required

An HTTP_E_* code, such as a client's error.

returnsstring

The code's name, "none" for HTTP_E_NONE, or "unknown" for a number that is no code.

httpNoProxyMatch

Whether noProxy, a comma-separated list, names host on port: an entry is * (every host), a name that matches the host or any domain under it (with or without a leading dot), or an IP address that matches only itself; any of them may end in :port, and an IPv6 address with a port is written in brackets. Case does not matter. Address ranges are not understood.

const httpNoProxyMatch: (
  noProxy: string,
  host: string,
  port: i32,
) => boolean
noProxystringrequired

The list, as NO_PROXY holds it; an entry that does not parse is skipped.

hoststringrequired

The request's host name or IP address; trailing dots do not matter.

porti32required

The port the request goes to.

returnsboolean

True when an entry names the host (and its port, if it has one), so the request goes direct.

httpProxyFor

The proxy a request to scheme://host:port goes through, given the three settings: httpsProxy for https, httpProxy for http, and "" for none or when noProxy names the host.

const httpProxyFor: (
  scheme: string,
  host: string,
  port: i32,
  httpProxy: string,
  httpsProxy: string,
  noProxy: string,
) => string
schemestringrequired

The request URL's scheme, lower-cased; one other than http or https gets no proxy.

hoststringrequired

The request URL's host, matched against noProxy.

porti32required

The port the request goes to, matched against an entry's :port.

httpProxystringrequired

The proxy URL for http requests, or "" for none.

httpsProxystringrequired

The proxy URL for https requests, or "" for none.

noProxystringrequired

The comma-separated hosts that go direct, as httpNoProxyMatch reads them.

returnsstring

The proxy URL the request goes through, or "" when it goes direct.

Classes

FetchError

Why an async fetch failed: an HTTP_E_* code and its sentence, the code's name first, as FetchSyncClient.errorText reads. The text never holds a URL's credentials or query.

class FetchError
Member
new FetchError(code: i32, text: string) A failure with its code and text.
code: i32 The HTTP_E_* code.
text: string The code's name, a colon and what went wrong, such as dns: example.invalid could not be looked up.

FetchPool

Connections an async fetch leaves open for the next one to the same place, held by the caller and handed in through RequestInit.pool. At most maxIdlePerOrigin wait per route and 64 in all; a kept connection the server closed, or sent a byte on unasked, is closed when it is next looked at. A fetch under an outbound policy never keeps or takes one, and nor does an https fetch: the pool holds sockets, not their TLS state.

class FetchPool
Member
new FetchPool(maxIdlePerOrigin: i32 = 4) An empty pool.
maxIdlePerOrigin: i32 The most idle connections kept for one route (origin, or proxy).
close(): void Closes every idle connection; the pool can still keep new ones.
idle(): i32 How many idle connections the pool holds.

FetchResponse

A response fetch answered, its body already read whole: the body is read before fetch answers, up to maxBodyBytes, so text() and bytes() are plain calls. names are lower-cased and in the order they arrived, beside values; url is the URL that answered.

class FetchResponse
Member
new FetchResponse() A response with no status, fields or body yet; fetch fills them in.
body: u8[] The body, de-chunked; empty for a HEAD or a status that has none.
names: string[] The header field names, lower-cased, in the order they arrived.
reason: string The reason phrase the server sent beside the status.
status: i32 The status code, such as 200.
url: string The URL that answered, which differs from the request's after a redirect.
values: string[] The header field values, each beside its name in names.
bytes(): u8[] The body's bytes.
header(name: string): string | null The first value of the field name, in any case.
text(): string The body as a string, its bytes unchanged: nothing checks that they are UTF-8.

FetchSyncClient

A client: the options, the proxy settings, and at most one open connection, kept between fetches. Not for two threads at once.

class FetchSyncClient
Member
new FetchSyncClient(options: HttpOptions) A client with no connection open yet, its proxy settings read from the environment when useEnvProxy is set.
error: i32 The last failure's HTTP_E_* code; HTTP_E_NONE after a fetch that answered.
errorText: string A sentence saying why the last failure happened, for a log; "" after a fetch that answered.
options: HttpOptions The options every fetch through this client follows.
reused: boolean Whether fd has carried a response before the one in flight.
close(): void Closes the connection; the client can still fetch, on a new one.
fetch(req: HttpRequest): HttpResponse | null Sends req and answers the response once its head is in, following redirects as the options say; null on a failure, with error and errorText set. The body is read through the response's body before the next fetch.
setProxies(httpProxy: string, httpsProxy: string, noProxy: string): void Replaces the proxy settings, as the three variables would set them; "" for none.

HttpBodyReader

A response's body, read as it arrives: de-chunked, bounded by maxBodyBytes, and ending where the framing says it ends, so a cut-short body is an error and never taken for a whole one. It reads through the client that made it, and only until that client starts its next fetch.

class HttpBodyReader
Member
new HttpBodyReader(client: FetchSyncClient, generation: i32) A reader for the body of one response.
client: FetchSyncClient The client that made the response; it carries the connection the body arrives on.
read(buf: u8[], off: i32, n: i32): i32 Up to n body bytes into buf[off ..]: how many (at least one), 0 at the end of the body, or a negated HTTP_E_* code, the client's errorText saying why. It waits for at least one byte, no more, so an event stream is read event by event. A window outside buf panics; an empty one answers 0 at once, reading nothing, so 0 means the end only when n is positive.
readAll(limit: i32): u8[] | null The rest of the body in one array, or null on an error or once it passes limit bytes (HTTP_E_BODY_TOO_LARGE, and the connection is closed).

HttpOptions

How a client behaves. Every field has a default meant for a CLI talking to an API; raise maxBodyBytes to download something large, and set readTimeoutMs to 0 for an event stream that may be quiet for a long time.

class HttpOptions
Member
new HttpOptions()
connectTimeoutMs: i32 The budget for the lookup and the connect together; 0 or less for none but the total.
crossOriginRedirects: boolean Let a followed redirect go to another origin (see the module comment).
distrustSpkiSha256: u8[][] | null Keys no https server's chain may hold, each the SHA-256 of a certificate's subjectPublicKeyInfo DER, 32 bytes: a chain with a leaf, intermediate or root whose key is listed fails with HTTP_E_CERTIFICATE (tlsClientDistrustSpki). Null, the default, for none; a hash of another length fails every https fetch rather than being skipped.
dns: DnsConfig | null The resolver settings a name is looked up with, as dnsParseResolvConf fills them; null reads /etc/resolv.conf. /etc/hosts is consulted either way. Only read, so one config can serve many fetches.
followRedirects: boolean Follow a redirect rather than answer it; only to the same origin unless crossOriginRedirects is set.
maxBodyBytes: i64 256 MiB by default; a body past it fails with HTTP_E_BODY_TOO_LARGE.
maxHeaderBytes: i32 The status line and the header section, each at most this.
maxRedirects: i32 The most redirect hops one fetch follows; past them the fetch fails.
outbound: OutboundPolicy | null For a URL the program did not choose: only the addresses and hosts the policy allows are reached, no proxy is used, and its caps apply (see nish/net/outbound). Null for none.
readTimeoutMs: i32 The longest wait for the server to send or take a byte; 0 or less for none.
rootsPemPath: string | null A PEM file of the roots an https server's chain must end at; null for the system's.
totalTimeoutMs: i32 The budget for a whole fetch, its body included; 0 or less for none.
useEnvProxy: boolean Read the proxy settings from the environment (see the module comment).

HttpRequest

One request: a method, an absolute http:// or https:// URL, header fields in the order they are sent, and an optional body. A Host field is added from the URL unless the caller gives one, and Content-Length from the body, so a caller never frames the body itself. Userinfo in the URL is not sent anywhere: a program that means to authenticate sets its own Authorization field, so a password is never sent by accident.

class HttpRequest
Member
new HttpRequest(method: string, url: string) A request with no header fields and no body.
body: u8[] | null Null for no body at all; an empty array sends Content-Length: 0.
method: string The request method, sent as given, such as GET or POST.
names: string[] The header field names, in the order they are sent.
url: string The absolute URL the request is sent to.
values: string[] The header field values, each beside its name in names.
header(name: string, value: string): void Appends one header field; nothing is checked until the request is written.

HttpResponse

A response's head, as the server sent it, and a reader for its body. url is the URL that answered, which differs from the request's after a redirect. names are lower-cased and in the order they arrived, beside values.

class HttpResponse
Member
new HttpResponse(body: HttpBodyReader) A response with no status and no fields yet; the client fills them in.
body: HttpBodyReader The reader for the response's body.
names: string[] The header field names, lower-cased, in the order they arrived.
reason: string The reason phrase the server sent beside the status.
status: i32 The status code, such as 200.
url: string The URL that answered, which differs from the request's after a redirect.
values: string[] The header field values, each beside its name in names.
header(name: string): string | null The first value of the field name, in any case.

RequestInit

What an async fetch sends and how it behaves. options are HttpOptions as fetchSync reads them, with the same deadlines: connectTimeoutMs bounds the lookup and the connect, readTimeoutMs each wait on the server, and totalTimeoutMs the whole fetch, each wait parked with readableWithin or writableWithin until the nearest of them. A fetch whose task is cancelled fails with HTTP_E_CANCELLED. pool keeps connections for later fetches; null closes each one when its fetch is done.

class RequestInit
Member
new RequestInit() A GET with no fields, no body, default options and no pool.
body: u8[] | null Null for no body at all; an empty array sends Content-Length: 0.
method: string The request method, sent as given; GET by default.
names: string[] The header field names, in the order they are sent.
options: HttpOptions The limits, policy and redirect and proxy settings (see above for the timeouts).
pool: FetchPool | null Where idle connections are kept between fetches; null for none.
values: string[] The header field values, each beside its name in names.
header(name: string, value: string): RequestInit Appends one header field; nothing is checked until the request is written.

Constants

17

HTTP_E_BLOCKED: i32

The outbound policy refused the URL's host or every address it resolved to.

10

HTTP_E_BODY_TOO_LARGE: i32

A body over maxBodyBytes, or over the limit given to readAll.

18

HTTP_E_CANCELLED: i32

The async fetch's task was cancelled (nish:async's cancel, or a deadline set with cancelAfter).

13

HTTP_E_CERTIFICATE: i32

The server's certificate chain was not accepted for its host.

4

HTTP_E_CONNECT_TIMEOUT: i32

No address of the host answered within connectTimeoutMs.

2

HTTP_E_DNS: i32

The host's name could not be looked up, or has no address.

15

HTTP_E_IO: i32

A socket call failed, or the server closed the connection before it answered.

0

HTTP_E_NONE: i32

No error.

7

HTTP_E_PROTOCOL: i32

The response broke HTTP/1.1; errorText names the parser's code.

16

HTTP_E_PROXY: i32

A proxy setting that is not an http:// URL with a host.

11

HTTP_E_PROXY_REFUSED: i32

The proxy answered its CONNECT with something other than 2xx.

5

HTTP_E_READ_TIMEOUT: i32

The server sent nothing, or took nothing, for readTimeoutMs.

9

HTTP_E_REDIRECT_REFUSED: i32

A redirect to another origin, from https to http, or to a URL that is not http or https.

3

HTTP_E_REFUSED: i32

Every address of the host refused the connection, or could not be reached.

14

HTTP_E_REQUEST: i32

The request itself was refused before a byte was sent: a bad method, header or target.

12

HTTP_E_TLS: i32

The TLS handshake or a TLS record failed.

8

HTTP_E_TOO_MANY_REDIRECTS: i32

A redirect past maxRedirects.

6

HTTP_E_TOTAL_TIMEOUT: i32

The fetch ran past totalTimeoutMs.

1

HTTP_E_URL: i32

The request's URL is not an absolute http or https URL.