nish/net/fetch
An HTTP/1.1 client. fetchSync(request, options) is the blocking one for command-line programs: it dials, sends, and answers the response's head with a reader for its body, and a FetchSyncClient does the same while keeping one connection alive between fetches. await fetch(url, init) is the async one: it parks its task on nish:async instead of the thread, so awaitAll(fetch(a), fetch(b)) overlaps the two.
import {
FetchError,
FetchPool,
FetchResponse,
FetchSyncClient,
HTTP_E_BLOCKED,
HTTP_E_BODY_TOO_LARGE,
HTTP_E_CANCELLED,
HTTP_E_CERTIFICATE,
HTTP_E_CONNECT_TIMEOUT,
HTTP_E_DNS,
HTTP_E_IO,
HTTP_E_NONE,
HTTP_E_PROTOCOL,
HTTP_E_PROXY,
HTTP_E_PROXY_REFUSED,
HTTP_E_READ_TIMEOUT,
HTTP_E_REDIRECT_REFUSED,
HTTP_E_REFUSED,
HTTP_E_REQUEST,
HTTP_E_TLS,
HTTP_E_TOO_MANY_REDIRECTS,
HTTP_E_TOTAL_TIMEOUT,
HTTP_E_URL,
HttpBodyReader,
HttpOptions,
HttpRequest,
HttpResponse,
RequestInit,
fetch,
fetchSync,
httpChooseProxy,
httpErrorName,
httpNoProxyMatch,
httpProxyFor,
} from "nish/net/fetch";
import { fetch } from "nish/net/fetch";
const r = await fetch("http://127.0.0.1:8080/health");
if (r.isOk()) {
const body: string = r.value.text();
}
fetch
Fetches url without blocking the process: the task waits on the executor while the connection is made, the request written and the response read, so other tasks run meanwhile and awaitAll(fetch(a), fetch(b)) overlaps the two.
fetchSync
One fetch on a client of its own, closed once the body has been read or the fetch failed.
httpChooseProxy
A proxy variable's value: lower when it is set and not empty, else upper when upperAllowed, else "".
httpErrorName
A short name for an HTTP_E_* code, for messages: "dns", "read timeout", ...
httpNoProxyMatch
Whether noProxy, a comma-separated list, names host on port: an entry is * (every host), a name that matches the host or any domain under it (with or without a leading dot), or an IP address that matches only itself; any of them may end in :port, and an IPv6 address with a port is written in brackets.
httpProxyFor
The proxy a request to scheme://host:port goes through, given the three settings: httpsProxy for https, httpProxy for http, and "" for none or when noProxy names the host.
FetchError
Why an async fetch failed: an HTTP_E_* code and its sentence, the code's name first, as FetchSyncClient.errorText reads.
FetchPool
Connections an async fetch leaves open for the next one to the same place, held by the caller and handed in through RequestInit.pool.
FetchResponse
A response fetch answered, its body already read whole: the body is read before fetch answers, up to maxBodyBytes, so text() and bytes() are plain calls.
FetchSyncClient
A client: the options, the proxy settings, and at most one open connection, kept between fetches.
HttpBodyReader
A response's body, read as it arrives: de-chunked, bounded by maxBodyBytes, and ending where the framing says it ends, so a cut-short body is an error and never taken for a whole one.
HttpOptions
How a client behaves.
HttpRequest
One request: a method, an absolute http:// or https:// URL, header fields in the order they are sent, and an optional body.
HttpResponse
A response's head, as the server sent it, and a reader for its body.
RequestInit
What an async fetch sends and how it behaves.
The async fetch answers a Result: a FetchResponse whose body was
read whole before it answered (up to maxBodyBytes, 256 MiB by default,
else HTTP_E_BODY_TOO_LARGE), or a FetchError with the HTTP_E_* code.
The name lookup, the connect, the TLS handshake, every write and every
read wait on the executor, so a slow resolver or server stalls only its own
task. An https:// URL is checked as fetchSync checks it: the caller's
roots or the system's, the host name and the wall clock, with no setting
that skips it. A FetchPool handed in through RequestInit.pool keeps
idle http connections per route, capped; an https one is closed after its
fetch, never pooled.
Policy, proxies, redirects and error codes are the shared code below, so
both clients behave alike. It does not run under --emit-ts.
import { HttpOptions, HttpRequest, HttpResponse, fetchSync } from "nish/net/fetch";
const req = new HttpRequest("GET", "http://example.com/");
req.header("Accept", "application/json");
const res: HttpResponse | null = fetchSync(req, new HttpOptions());
if (res !== null) {
const body: u8[] | null = res.body.readAll(1048576);
}
It blocks the thread
It blocks the thread at every step, the way a CLI wants it to: the
lookup, the connect, each write and each read. A server, which must not
stall every connection for one slow peer, drives nish/net/http1-client-codec's
parser from its own loop instead. Every wait is bounded: connectTimeoutMs
for the lookup and the connect (one budget across every address the name
has), readTimeoutMs for each wait on the peer, and totalTimeoutMs, when
set, for the whole fetch, redirects and body included, so a server that
drips one byte a second cannot hold a fetch forever.
Framing is the parser's
The head is written by http1RequestHead and
every byte read goes through Http1ResponseParser, so a value holding a CR
or LF is refused rather than sent, and every response-smuggling shape is
refused with the parser's own code (see docs/security/http1-client-codec.md).
Keep-alive
Keep-alive holds one connection, to the last origin (or proxy) used. A fetch to the same place reuses it once the last body has been read to its end; a fetch while a body is unread closes it, and the old reader answers an error. A reused connection the server closed while it sat idle is the one failure a client cannot avoid, so a request with an idempotent method that fails on it before any response byte arrives is sent once more on a fresh connection (RFC 9110 §9.2.2). A POST is never sent twice.
Redirects
Redirects are not followed unless followRedirects is set, and then
only to the same origin unless crossOriginRedirects is set too. A hop to
another origin drops Authorization, Proxy-Authorization, Cookie and
the caller's Host; a hop from https to http is always refused; 303 turns
any method but HEAD into a GET without a body, as do 301 and 302 after a
POST, and 307 and 308 keep the method and the body (RFC 9110 §15.4). Past
maxRedirects hops the fetch fails.
Proxies
Proxies come from the environment when useEnvProxy is set:
http_proxy (or HTTP_PROXY), https_proxy (or HTTPS_PROXY) and
no_proxy (or NO_PROXY), the lower-case name first. HTTP_PROXY is
ignored when REQUEST_METHOD is set, since a CGI program finds a request's
Proxy: header there under that name (the "httpoxy" hole). Only an
http:// proxy is accepted. An http request goes to it in absolute form; an
https one goes through a CONNECT tunnel.
Every failure answers null with a distinct HTTP_E_* code in the client's
error and a sentence in errorText; a body reader answers the code
negated.
Private helpers share the importing program's flat symbol namespace
(docs/wp26-stdlib.md §3e), which is why each one carries the module's name.
Written from RFC 9110 and RFC 9112, not ported from another implementation.
Functions
fetch
Fetches url without blocking the process: the task waits on the
executor while the connection is made, the request written and the
response read, so other tasks run meanwhile and awaitAll(fetch(a), fetch(b)) overlaps the two. The name lookup does not block either: it
runs on non-blocking sockets within connectTimeoutMs. An https URL gets
the TLS handshake and checks fetchSync gives it. The body is read whole,
up to maxBodyBytes, before the answer comes. Policy, proxy, redirect and
error rules are fetchSync's.
const fetch: (
url: string,
init: RequestInit | null = null,
) => Promise<Result<FetchResponse, FetchError>>
urlstringrequiredThe absolute http:// or https:// URL to fetch.
initRequestInit | nullThe method, fields, body, options and pool; null for a plain GET.
returnsPromise<Result<FetchResponse, FetchError>>The response with its body, or the failure: never a panic.
fetchSync
One fetch on a client of its own, closed once the body has been read or
the fetch failed. Null on a failure, with no way to say why: a program
that reports its errors fetches through a FetchSyncClient.
const fetchSync: (
req: HttpRequest,
options: HttpOptions,
) => HttpResponse | null
reqHttpRequestrequiredThe request to send, as a FetchSyncClient takes it.
optionsHttpOptionsrequiredThe client's limits, timeouts, TLS trust and proxies for this one fetch.
returnsHttpResponse | nullThe response, the last one when redirects were followed, or null on any failure.
httpChooseProxy
A proxy variable's value: lower when it is set and not empty, else
upper when upperAllowed, else "". HTTP_PROXY is not allowed under CGI.
const httpChooseProxy: (
lower: string | null,
upper: string | null,
upperAllowed: boolean,
) => string
lowerstring | nullrequiredThe lower-case variable's value, such as http_proxy's, or null when it is unset.
upperstring | nullrequiredThe upper-case variable's value, such as HTTP_PROXY's, or null when it is unset.
upperAllowedbooleanrequiredWhether the upper-case variable may be read: false for HTTP_PROXY under CGI, where a request's Proxy header sets it.
returnsstringThe chosen variable's value, a proxy URL or for no_proxy a comma-separated host list, or "" when neither applies.
httpErrorName
A short name for an HTTP_E_* code, for messages: "dns", "read timeout",
... A fetch's errorText starts with it.
const httpErrorName: (code: i32) => string
codei32requiredAn HTTP_E_* code, such as a client's error.
returnsstringThe code's name, "none" for HTTP_E_NONE, or "unknown" for a number that is no code.
httpNoProxyMatch
Whether noProxy, a comma-separated list, names host on port: an entry
is * (every host), a name that matches the host or any domain under it
(with or without a leading dot), or an IP address that matches only itself;
any of them may end in :port, and an IPv6 address with a port is written
in brackets. Case does not matter. Address ranges are not understood.
const httpNoProxyMatch: (
noProxy: string,
host: string,
port: i32,
) => boolean
noProxystringrequiredThe list, as NO_PROXY holds it; an entry that does not parse is skipped.
hoststringrequiredThe request's host name or IP address; trailing dots do not matter.
porti32requiredThe port the request goes to.
returnsbooleanTrue when an entry names the host (and its port, if it has one), so the request goes direct.
httpProxyFor
The proxy a request to scheme://host:port goes through, given the three
settings: httpsProxy for https, httpProxy for http, and "" for none or
when noProxy names the host.
const httpProxyFor: (
scheme: string,
host: string,
port: i32,
httpProxy: string,
httpsProxy: string,
noProxy: string,
) => string
schemestringrequiredThe request URL's scheme, lower-cased; one other than http or https gets no proxy.
hoststringrequiredThe request URL's host, matched against noProxy.
porti32requiredThe port the request goes to, matched against an entry's :port.
httpProxystringrequiredThe proxy URL for http requests, or "" for none.
httpsProxystringrequiredThe proxy URL for https requests, or "" for none.
noProxystringrequiredThe comma-separated hosts that go direct, as httpNoProxyMatch reads them.
returnsstringThe proxy URL the request goes through, or "" when it goes direct.
Classes
FetchError
Why an async fetch failed: an HTTP_E_* code and its sentence, the
code's name first, as FetchSyncClient.errorText reads. The text never
holds a URL's credentials or query.
class FetchError
| Member | |
|---|---|
new FetchError(code: i32, text: string) |
A failure with its code and text. |
code: i32 |
The HTTP_E_* code. |
text: string |
The code's name, a colon and what went wrong, such as dns: example.invalid could not be looked up. |
FetchPool
Connections an async fetch leaves open for the next one to the same
place, held by the caller and handed in through RequestInit.pool. At
most maxIdlePerOrigin wait per route and 64 in all; a kept connection
the server closed, or sent a byte on unasked, is closed when it is next
looked at. A fetch under an outbound policy never keeps or takes one, and
nor does an https fetch: the pool holds sockets, not their TLS state.
class FetchPool
| Member | |
|---|---|
new FetchPool(maxIdlePerOrigin: i32 = 4) |
An empty pool. |
maxIdlePerOrigin: i32 |
The most idle connections kept for one route (origin, or proxy). |
close(): void |
Closes every idle connection; the pool can still keep new ones. |
idle(): i32 |
How many idle connections the pool holds. |
FetchResponse
A response fetch answered, its body already read whole: the body is
read before fetch answers, up to maxBodyBytes, so text() and
bytes() are plain calls. names are lower-cased and in the order they
arrived, beside values; url is the URL that answered.
class FetchResponse
| Member | |
|---|---|
new FetchResponse() |
A response with no status, fields or body yet; fetch fills them in. |
body: u8[] |
The body, de-chunked; empty for a HEAD or a status that has none. |
names: string[] |
The header field names, lower-cased, in the order they arrived. |
reason: string |
The reason phrase the server sent beside the status. |
status: i32 |
The status code, such as 200. |
url: string |
The URL that answered, which differs from the request's after a redirect. |
values: string[] |
The header field values, each beside its name in names. |
bytes(): u8[] |
The body's bytes. |
header(name: string): string | null |
The first value of the field name, in any case. |
text(): string |
The body as a string, its bytes unchanged: nothing checks that they are UTF-8. |
FetchSyncClient
A client: the options, the proxy settings, and at most one open connection, kept between fetches. Not for two threads at once.
class FetchSyncClient
| Member | |
|---|---|
new FetchSyncClient(options: HttpOptions) |
A client with no connection open yet, its proxy settings read from the environment when useEnvProxy is set. |
error: i32 |
The last failure's HTTP_E_* code; HTTP_E_NONE after a fetch that answered. |
errorText: string |
A sentence saying why the last failure happened, for a log; "" after a fetch that answered. |
options: HttpOptions |
The options every fetch through this client follows. |
reused: boolean |
Whether fd has carried a response before the one in flight. |
close(): void |
Closes the connection; the client can still fetch, on a new one. |
fetch(req: HttpRequest): HttpResponse | null |
Sends req and answers the response once its head is in, following redirects as the options say; null on a failure, with error and errorText set. The body is read through the response's body before the next fetch. |
setProxies(httpProxy: string, httpsProxy: string, noProxy: string): void |
Replaces the proxy settings, as the three variables would set them; "" for none. |
HttpBodyReader
A response's body, read as it arrives: de-chunked, bounded by
maxBodyBytes, and ending where the framing says it ends, so a cut-short
body is an error and never taken for a whole one. It reads through the
client that made it, and only until that client starts its next fetch.
class HttpBodyReader
| Member | |
|---|---|
new HttpBodyReader(client: FetchSyncClient, generation: i32) |
A reader for the body of one response. |
client: FetchSyncClient |
The client that made the response; it carries the connection the body arrives on. |
read(buf: u8[], off: i32, n: i32): i32 |
Up to n body bytes into buf[off ..]: how many (at least one), 0 at the end of the body, or a negated HTTP_E_* code, the client's errorText saying why. It waits for at least one byte, no more, so an event stream is read event by event. A window outside buf panics; an empty one answers 0 at once, reading nothing, so 0 means the end only when n is positive. |
readAll(limit: i32): u8[] | null |
The rest of the body in one array, or null on an error or once it passes limit bytes (HTTP_E_BODY_TOO_LARGE, and the connection is closed). |
HttpOptions
How a client behaves. Every field has a default meant for a CLI talking to
an API; raise maxBodyBytes to download something large, and set
readTimeoutMs to 0 for an event stream that may be quiet for a long time.
class HttpOptions
| Member | |
|---|---|
new HttpOptions() |
|
connectTimeoutMs: i32 |
The budget for the lookup and the connect together; 0 or less for none but the total. |
crossOriginRedirects: boolean |
Let a followed redirect go to another origin (see the module comment). |
distrustSpkiSha256: u8[][] | null |
Keys no https server's chain may hold, each the SHA-256 of a certificate's subjectPublicKeyInfo DER, 32 bytes: a chain with a leaf, intermediate or root whose key is listed fails with HTTP_E_CERTIFICATE (tlsClientDistrustSpki). Null, the default, for none; a hash of another length fails every https fetch rather than being skipped. |
dns: DnsConfig | null |
The resolver settings a name is looked up with, as dnsParseResolvConf fills them; null reads /etc/resolv.conf. /etc/hosts is consulted either way. Only read, so one config can serve many fetches. |
followRedirects: boolean |
Follow a redirect rather than answer it; only to the same origin unless crossOriginRedirects is set. |
maxBodyBytes: i64 |
256 MiB by default; a body past it fails with HTTP_E_BODY_TOO_LARGE. |
maxHeaderBytes: i32 |
The status line and the header section, each at most this. |
maxRedirects: i32 |
The most redirect hops one fetch follows; past them the fetch fails. |
outbound: OutboundPolicy | null |
For a URL the program did not choose: only the addresses and hosts the policy allows are reached, no proxy is used, and its caps apply (see nish/net/outbound). Null for none. |
readTimeoutMs: i32 |
The longest wait for the server to send or take a byte; 0 or less for none. |
rootsPemPath: string | null |
A PEM file of the roots an https server's chain must end at; null for the system's. |
totalTimeoutMs: i32 |
The budget for a whole fetch, its body included; 0 or less for none. |
useEnvProxy: boolean |
Read the proxy settings from the environment (see the module comment). |
HttpRequest
One request: a method, an absolute http:// or https:// URL, header
fields in the order they are sent, and an optional body. A Host field is
added from the URL unless the caller gives one, and Content-Length from
the body, so a caller never frames the body itself. Userinfo in the URL is
not sent anywhere: a program that means to authenticate sets its own
Authorization field, so a password is never sent by accident.
class HttpRequest
| Member | |
|---|---|
new HttpRequest(method: string, url: string) |
A request with no header fields and no body. |
body: u8[] | null |
Null for no body at all; an empty array sends Content-Length: 0. |
method: string |
The request method, sent as given, such as GET or POST. |
names: string[] |
The header field names, in the order they are sent. |
url: string |
The absolute URL the request is sent to. |
values: string[] |
The header field values, each beside its name in names. |
header(name: string, value: string): void |
Appends one header field; nothing is checked until the request is written. |
HttpResponse
A response's head, as the server sent it, and a reader for its body. url
is the URL that answered, which differs from the request's after a redirect.
names are lower-cased and in the order they arrived, beside values.
class HttpResponse
| Member | |
|---|---|
new HttpResponse(body: HttpBodyReader) |
A response with no status and no fields yet; the client fills them in. |
body: HttpBodyReader |
The reader for the response's body. |
names: string[] |
The header field names, lower-cased, in the order they arrived. |
reason: string |
The reason phrase the server sent beside the status. |
status: i32 |
The status code, such as 200. |
url: string |
The URL that answered, which differs from the request's after a redirect. |
values: string[] |
The header field values, each beside its name in names. |
header(name: string): string | null |
The first value of the field name, in any case. |
RequestInit
What an async fetch sends and how it behaves. options are
HttpOptions as fetchSync reads them, with the same deadlines:
connectTimeoutMs bounds the lookup and the connect, readTimeoutMs
each wait on the server, and totalTimeoutMs the whole fetch, each wait
parked with readableWithin or writableWithin until the nearest of
them. A fetch whose task is cancelled fails with HTTP_E_CANCELLED.
pool keeps connections for later fetches; null closes each one when
its fetch is done.
class RequestInit
| Member | |
|---|---|
new RequestInit() |
A GET with no fields, no body, default options and no pool. |
body: u8[] | null |
Null for no body at all; an empty array sends Content-Length: 0. |
method: string |
The request method, sent as given; GET by default. |
names: string[] |
The header field names, in the order they are sent. |
options: HttpOptions |
The limits, policy and redirect and proxy settings (see above for the timeouts). |
pool: FetchPool | null |
Where idle connections are kept between fetches; null for none. |
values: string[] |
The header field values, each beside its name in names. |
header(name: string, value: string): RequestInit |
Appends one header field; nothing is checked until the request is written. |
Constants
17
HTTP_E_BLOCKED: i32
The outbound policy refused the URL's host or every address it resolved to.
10
HTTP_E_BODY_TOO_LARGE: i32
A body over maxBodyBytes, or over the limit given to readAll.
18
HTTP_E_CANCELLED: i32
The async fetch's task was cancelled (nish:async's cancel, or a deadline set with cancelAfter).
13
HTTP_E_CERTIFICATE: i32
The server's certificate chain was not accepted for its host.
4
HTTP_E_CONNECT_TIMEOUT: i32
No address of the host answered within connectTimeoutMs.
2
HTTP_E_DNS: i32
The host's name could not be looked up, or has no address.
15
HTTP_E_IO: i32
A socket call failed, or the server closed the connection before it answered.
0
HTTP_E_NONE: i32
No error.
7
HTTP_E_PROTOCOL: i32
The response broke HTTP/1.1; errorText names the parser's code.
16
HTTP_E_PROXY: i32
A proxy setting that is not an http:// URL with a host.
11
HTTP_E_PROXY_REFUSED: i32
The proxy answered its CONNECT with something other than 2xx.
5
HTTP_E_READ_TIMEOUT: i32
The server sent nothing, or took nothing, for readTimeoutMs.
9
HTTP_E_REDIRECT_REFUSED: i32
A redirect to another origin, from https to http, or to a URL that is not http or https.
3
HTTP_E_REFUSED: i32
Every address of the host refused the connection, or could not be reached.
14
HTTP_E_REQUEST: i32
The request itself was refused before a byte was sent: a bad method, header or target.
12
HTTP_E_TLS: i32
The TLS handshake or a TLS record failed.
8
HTTP_E_TOO_MANY_REDIRECTS: i32
A redirect past maxRedirects.
6
HTTP_E_TOTAL_TIMEOUT: i32
The fetch ran past totalTimeoutMs.
1
HTTP_E_URL: i32
The request's URL is not an absolute http or https URL.

