HTTP client
Two modules speak HTTP/1.1 as a client. nish/net/fetch is the one a
program wants: fetchSync takes a URL and blocks until the response is in,
with a timeout on every wait and a cap on every body, and await fetch(url)
does the same without blocking the process, so other tasks run while it waits
(Fetching without blocking).
nish/net/http1-client-codec, below it, is the protocol alone: it writes a
request's head and reads the response, bytes in and events out, and opens no
socket, for a program that runs its own loop. Both are imported like any other
library module (export and import).
Fetching a URL
import { HttpOptions, HttpRequest, HttpResponse, fetchSync } from "nish/net/fetch";
export const main = (): i32 => {
const req = new HttpRequest("GET", "http://example.com/");
req.header("Accept", "text/html");
const options = new HttpOptions();
options.totalTimeoutMs = 20000;
const res: HttpResponse | null = fetchSync(req, options);
if (res === null) {
console.error("the fetch failed");
return 1;
}
const type = res.header("content-type");
console.log(`${res.status} ${res.reason}, ${type === null ? "no type" : type}`);
const body: u8[] | null = res.body.readAll(1048576);
if (body === null) {
console.error("the body failed or was larger than 1 MiB");
return 1;
}
console.log(String.fromBytes(body, 0, toI32(body.length)));
return 0;
};
fetchSync looks the host up, connects, sends the request and answers once
the response's head is in: status, reason, the url that answered and
header(name), in any case. The body is read through res.body:
readAll(limit) answers it whole, or null past limit or on a failure, and
read(buf, off, n) answers the next bytes as they arrive, 0 at the end and a
negative code on a failure, for a stream you want as it comes; a read of 0
bytes answers 0 at once.
Once the body ends or fails, fetchSync closes its connection; read it to the
end, or, when you only want the status, close it with
res.body.client.close(), since an unread body holds its socket open until the
program exits. A request carries a body when you set req.body, with its
Content-Length sent for you. A header value holding a line break, or a method
that is not a token, is refused before a byte is sent.
Knowing why it failed
fetchSync answers null with no reason. An FetchSyncClient does the same work,
says why in error (an HTTP_E_* code; httpErrorName names it) and
errorText (a sentence, such as could not connect to example.com port 80 (errno 111)), and keeps the connection alive between fetches to the same
place.
A URL that errorText names is only its scheme, host, port and path,
with ?... where a query or fragment was left out, and a URL that does not
parse is not echoed at all. The URL is named as it parses: in
http://admin:8080/s3cret@example.com/ the 8080 is a port and the rest a
path, which is echoed, so put credentials in a header, not in a URL:
import { FetchSyncClient, HttpOptions, HttpRequest, HttpResponse, httpErrorName } from "nish/net/fetch";
export const main = (): i32 => {
const client = new FetchSyncClient(new HttpOptions());
const res: HttpResponse | null = client.fetch(new HttpRequest("GET", "http://example.com/"));
if (res === null) {
console.error(`${httpErrorName(client.error)}: ${client.errorText}`);
client.close();
return 1;
}
console.log(`${res.status}`);
res.body.readAll(1048576);
client.close();
return 0;
};
Read each body to its end before the next fetch: the next fetch on a connection with an unread body closes it, and the old body's reader then answers an error rather than the next response's bytes. A connection is kept only when nothing is left on it past the response: a server that sends more than was asked for, or speaks while the connection sits idle, has it closed, and the next fetch opens a new one.
A kept connection the server has closed is seen before the request is sent, and the request goes on a fresh one. If the server closes it under the request, before a byte of the answer, the request is sent once more on a fresh one for a GET, HEAD, OPTIONS, TRACE, PUT or DELETE, never for a POST; a request whose answer was cut off part way, or that was never answered, is not sent again.
Fetching without blocking
In an async function, await fetch(url, init) fetches without holding up
the process: while it looks the name up, connects, runs the TLS handshake of
an https URL, writes and waits for the answer, the program's other tasks run
(Async functions). Two fetches under
awaitAll are in flight together:
import { awaitAll } from "nish:async";
import { FetchPool, RequestInit, fetch } from "nish/net/fetch";
const title = async (url: string, pool: FetchPool): Promise<string> => {
const init = new RequestInit();
init.header("Accept", "text/plain");
init.pool = pool;
const r = await fetch(url, init);
if (r.isErr()) {
return r.error.text;
}
return `${r.value.status}: ${r.value.text()}`;
};
export const main = async (): Promise<i32> => {
const pool = new FetchPool();
const [a, b] = await awaitAll(
title("http://127.0.0.1:8080/a", pool),
title("http://127.0.0.1:8080/b", pool)
);
console.log(a);
console.log(b);
pool.close();
return 0;
};
fetch answers a Result: a FetchResponse or a FetchError, whose code
is the HTTP_E_* code fetchSync's client would have set and whose text
is its name and sentence, such as refused: could not connect to 127.0.0.1 port 8080 (errno 111). It never answers null and never panics. A 404 or a
500 is a response, not an error: read status.
The body is read whole before fetch answers, up to the options'
maxBodyBytes (256 MiB unless you lower it; past it the fetch fails with
HTTP_E_BODY_TOO_LARGE), so text() and bytes() are plain calls, not
awaited. Set maxBodyBytes to what you expect from a server you do not run.
A body is held in one array, so even a raised maxBodyBytes stops just under
2 GiB.
What a fetch allocates is freed when the async function that awaited it
returns. A program that fetches in a long loop should await each fetch in a
function of its own, as title above is, so each response goes when that
function ends. A loop that awaits fetch directly in main keeps every
response until main returns.
header(name) reads a field in any case, url is the URL that answered.
RequestInit holds the request: method (GET unless set), fields added
with header(name, value), body (a u8[], with its Content-Length sent
for you), options (an HttpOptions) and pool. Pass null, or nothing, for a
plain GET. The options mean what they mean to fetchSync, redirects,
proxies and the outbound policy
included, except for the limits on the waits:
-
Each wait on the socket is bounded by
readTimeoutMs, and by what is left oftotalTimeoutMs, whichever ends first: a server that never sends another byte, or never answers the TLS handshake, fails the fetch withHTTP_E_READ_TIMEOUTorHTTP_E_TOTAL_TIMEOUT. -
A cancelled task's fetch fails with
HTTP_E_CANCELLEDat its next wait, whethercancel,cancelAfteror the server running the task (std/net/serve) cancelled it. -
connectTimeoutMsbounds the name lookup, which parks only its own task: the query goes out on a non-blocking socket, with the servers, tries and timeouts of/etc/resolv.conf(or ofoptions.dns), and the other tasks run until it answers. An address literal,localhostand a name in/etc/hostsneed no query. A lookup still unanswered atconnectTimeoutMsfails withHTTP_E_CONNECT_TIMEOUT.
https is the same TLS 1.3 as fetchSync's, with the same certificate
checks, the same roots (rootsPemPath) and the same errors: a certificate
that is expired, for another name or from a root you do not trust is
HTTP_E_CERTIFICATE with the sentence fetchSync gives. Through a proxy it
is a CONNECT tunnel. An https connection is never kept in a FetchPool:
each https fetch runs its own handshake and closes its connection once the
body is read.
A FetchPool keeps connections open between fetches: hand the same pool
to each RequestInit, and a fetch to a place the pool holds an idle connection
for takes it rather than dialling.
A connection is kept only when its response was read to the end and nothing is
left on it, at most maxIdlePerOrigin (4 unless given to the constructor) for
one origin or proxy and 64 in all. A kept connection the server closed, or wrote
to unasked, is closed when it is next looked at, and a GET whose kept connection
drops before a byte of the answer is sent once more on a fresh one, as
fetchSync does. A fetch under an outbound policy neither keeps nor takes one.
pool.idle() counts the idle connections and pool.close() closes them;
without a pool, each fetch closes its own.
fetch does not run under --emit-ts: it waits on its socket with
nish:async's readable and writable, which nish --emit-ts refuses with
NL4017 (Async functions). A program that must also run as
TypeScript uses fetchSync.
Options
HttpOptions holds every limit, set before the fetch:
| Field | Default | What it bounds |
|---|---|---|
connectTimeoutMs |
10,000 | The lookup and the connect, across every address the name has |
readTimeoutMs |
30,000 | Each wait for the server to send or take a byte |
totalTimeoutMs |
0 (none) | The whole fetch, redirects and body included |
maxHeaderBytes |
65,536 | The status line, and the header section |
maxBodyBytes |
256 MiB | Any body, however it is framed |
followRedirects |
false | Whether a 3xx with a Location is followed |
maxRedirects |
5 | How many hops |
crossOriginRedirects |
false | Whether a hop may go to another origin |
useEnvProxy |
true | Whether the proxy variables are read |
rootsPemPath |
null | A PEM file of the roots an https server must chain to, in place of the system's |
distrustSpkiSha256 |
null | Keys no certificate of an https server's chain may hold, each the SHA-256 of a subjectPublicKeyInfo |
outbound |
null | The outbound policy for URLs your program did not choose |
dns |
null | The resolver settings a name is looked up with, a DnsConfig as dnsParseResolvConf fills it, in place of /etc/resolv.conf; /etc/hosts is read either way |
A server that sends its body a byte at a time, each inside readTimeoutMs,
holds a fetch until totalTimeoutMs: set it when the server is not yours.
The addresses a name has are tried one after another inside
connectTimeoutMs: an address that drops the connection attempt without an
answer uses up the whole budget, so the addresses after it are not tried and
the fetch fails with HTTP_E_CONNECT_TIMEOUT.
Redirects are followed only when you set followRedirects, and only
within the origin unless you set crossOriginRedirects too. A hop to another
origin drops Authorization, Proxy-Authorization and Cookie, but not a
credential you send in a field of your own (X-Api-Key): leave
crossOriginRedirects off for such a request. A hop from https to http is
always refused. 303 turns the request into a GET without its body, as do 301
and 302 after a POST; 307 and 308 keep both.
Proxies come from http_proxy, https_proxy and no_proxy (or their
upper-case names), unless useEnvProxy is off; client.setProxies sets them
directly. HTTP_PROXY is ignored when the program runs as a CGI script,
where a request's Proxy: header arrives under that name. Only an http://
proxy is used; an https request goes through it in a CONNECT tunnel. The
proxy's user name and password come from its URL; a Proxy-Authorization
you set yourself goes only to an http proxy, in place of the one from its URL,
and never to the server.
https is TLS 1.3, and the server's certificate is always checked: its
chain must end at a root you trust, it must be valid now by the wall clock,
and it must name the URL's host (or its IP address, for an IP literal). The
roots are read once per client, from rootsPemPath when you set it, else
from the files SSL_CERT_FILE and SSL_CERT_DIR name, else from the system's
bundle.
A certificate that is expired, for another name, or from a root you
do not trust is HTTP_E_CERTIFICATE, and so is a fetch when no root can be
read at all, refused before any connection is opened. There is no switch that
turns the check off: to talk to a server with its own certificate authority,
point rootsPemPath at that authority's root. A chain through a key you
list in distrustSpkiSha256 is HTTP_E_CERTIFICATE too, and a hash there
that is not 32 bytes fails every https fetch rather than being skipped
(Distrusting a key). A server that does not speak
TLS, and a body cut off without TLS's close_notify, are HTTP_E_TLS.
Revocation is not checked.
Fetching a URL someone else chose
A program that fetches a URL a user typed, a webhook a customer registered, or
an image a page links to is a way into its own network for whoever wrote that
URL. http://169.254.169.254/ reaches a cloud host's metadata service and the
credentials in it; http://127.0.0.1:6379/ reaches a database that trusts
local callers; a name that resolves to either does the same. Give such a
client an OutboundPolicy, from nish/net/outbound:
import { HTTP_E_BLOCKED, FetchSyncClient, HttpOptions, HttpRequest, HttpResponse } from "nish/net/fetch";
import { OutboundPolicy } from "nish/net/outbound";
export const main = (): i32 => {
const policy = new OutboundPolicy();
const options = new HttpOptions();
options.outbound = policy;
options.followRedirects = true;
options.crossOriginRedirects = true;
const client = new FetchSyncClient(options);
const res: HttpResponse | null = client.fetch(new HttpRequest("GET", "http://127.0.0.1/"));
if (res === null && client.error === HTTP_E_BLOCKED) {
console.log(client.errorText);
}
client.close();
return 0;
};
It prints blocked: 127.0.0.1 resolves to 127.0.0.1, which the outbound policy does not allow, and no connection was opened. That text names the address a name
resolved to, so log it, but do not hand it back to whoever supplied the URL:
it would tell them your internal addresses. Answer them with a fixed message.
Under a policy:
-
Only public addresses are reached. The name is looked up first, every
address that is loopback, private (
10/8,172.16/12,192.168/16,fc00::/7), link-local (169.254/16,fe80::/10), shared (100.64/10), multicast, documentation or otherwise reserved is dropped, and the client connects only to the addresses it kept, without looking the name up again. A name whose answer changes between the check and the connect cannot move the connection. An IPv4 address written inside IPv6 (::ffff:127.0.0.1,64:ff9b::7f00:1,2002:7f00:1::) is judged as the IPv4 address it holds; a NAT64 prefix of your network's own choosing is not recognised. A name with no public address isHTTP_E_BLOCKED. -
Every redirect hop is checked the same way, so a public URL that answers
with a redirect to
http://localhost/fails at the hop. -
No proxy is used. A proxy looks the name up itself, past the check, so
the proxy variables and
setProxiesare ignored. -
A fetch is capped at 5 redirects, 30 seconds and an 8 MiB body, or your
options' limits where they are lower. Change them on the policy:
maxRedirects,totalTimeoutMs(0 for your options' alone) andmaxBodyBytes.
Two methods change what gets through. policy.permit("10.20.0.0/16") lets
a prefix you trust through anyway, such as an internal API the program is
meant to reach (its IPv4-mapped, NAT64 and 6to4 forms too); it answers false for text that is not an address and a prefix
length.
policy.allowHost("api.example.com") limits the names a URL, or any
hop, may carry: once one is added, a host that is not on the list is
HTTP_E_BLOCKED before it is looked up. A name is matched exactly, in any
case and with or without its trailing dot; an entry with a leading dot,
.example.com, also allows every name under it. An allowed host must still
resolve to an allowed address.
Only this client takes a policy: a connection opened another way, with
tcpConnect or another protocol's client, has no address check. A name you
did not choose is still looked up, so its nameserver sees the query; an
allowlist is the way to refuse a name before that. Set the policy before the
client's first fetch: a connection the client keeps from a fetch made under
another policy is closed rather than reused, so changing it costs a new
connection.
The policy decides where a fetch may go; it does not make the response safe to trust. Read the body under your own limits, as for any server you do not run.
The protocol alone
nish/net/http1-client-codec is the client side of HTTP/1.1 without the socket: it
writes a request's head and reads the response, bytes in and events out. It
opens no socket and knows no TLS: your program sends the bytes the writer
answers, and feeds the parser whatever it receives, cut anywhere. The event
constants are shared with the server side and come from nish/net/http1-codec.
A first request
import { HTTP1_BODY, HTTP1_END, HTTP1_ERROR, HTTP1_HEAD, HTTP1_NEED_MORE, HTTP1_NO_BODY } from "nish/net/http1-codec";
import { Http1ResponseParser, http1RequestHead } from "nish/net/http1-client-codec";
const bytesOf = (text: string): u8[] => {
const out: u8[] = [];
const n: i32 = toI32(text.length);
for (let i: i32 = 0; i < n; i++) {
out.push(toU8(text.charCodeAt(i)));
}
return out;
};
// Hands the parser every event its input holds, and answers whether the response ended.
const drain = (p: Http1ResponseParser): boolean => {
let event: i32 = p.next();
while (event !== HTTP1_NEED_MORE) {
if (event === HTTP1_HEAD) {
const type = p.header("content-type");
console.log(`status ${p.status}, ${type === null ? "no type" : type}`);
} else if (event === HTTP1_BODY) {
console.log(`body: ${String.fromBytes(p.body, p.bodyOff, p.bodyOff + p.bodyLen)}`);
} else if (event === HTTP1_END) {
console.log(`end, keep-alive ${p.keepAlive()}`);
return true;
} else if (event === HTTP1_ERROR) {
console.log(`refused with ${p.error}: ${p.errorText}`);
return true;
}
event = p.next();
}
return false;
};
export const main = (): i32 => {
const head = http1RequestHead("GET", "/hello", ["Host", "Accept"], ["example.com", "text/plain"], toI64(HTTP1_NO_BODY));
if (head === null) {
return 1;
}
console.log(String.fromBytes(head, 0, toI32(head.length)));
// What a server might answer, arriving in two pieces.
const reply = bytesOf("HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 5\r\n\r\nhello");
const p = new Http1ResponseParser(1024, 16384, 64, toI64(1048576), 4096);
p.expect(false, false);
p.feed(reply, 0, 20);
drain(p);
p.feed(reply, 20, toI32(reply.length) - 20);
drain(p);
return 0;
};
http1RequestHead answers the request line and the header section, Host
first and your fields after it in their order, or null for anything it
refuses. The parser is made once for a connection with its limits, told what
the request was with expect, and fed; next() answers one event at a time
until it answers HTTP1_NEED_MORE, which asks for more input. The events and
the body are the same however the response is cut: the first drain above
sees nothing, since the header section is not whole yet.
The request head
The last argument says how the body is framed, and the writer adds the field for it, after yours:
bodyLength |
The head ends with |
|---|---|
n, 0 or more |
Content-Length: n |
HTTP1_CHUNKED (-1) |
Transfer-Encoding: chunked; write each piece with http1Chunk and the end with http1LastChunk, from nish/net/http1-codec |
HTTP1_NO_BODY (-2) |
neither, for a request without a body, such as a GET |
http1WriteRequestHead(out, at, method, target, names, values, bodyLength)
writes the same head into your own u8[] at at and answers the offset just
past it. It answers HTTP1_NO_ROOM when the head does not fit and
HTTP1_REFUSED for what http1RequestHead answers null for; either way it
writes nothing.
The writer refuses whatever would let one of your strings end a line or frame the body on its own:
- a method that is not a token;
-
a target that is not a form the method may send: a path starting with
/or an absolute URL,host:portforCONNECT,*forOPTIONS; a target holding a space, a control byte, a non-ASCII byte or a#(ahost:portsent withGETis not refused: it is a well-formed absolute URL whose scheme is the host); - a field name that is not a token, or a value holding CR, LF or another control byte;
- anything but exactly one
Hostfield; - your own
Content-LengthorTransfer-Encoding, whichbodyLengthwrites; - a
Connectionfield that listsupgrade; namesandvaluesof different lengths, or abodyLengthbelow -2.
The events
| Event | What it means |
|---|---|
HTTP1_INFORMATIONAL |
an interim 1xx response was read; status and the fields are its, and the final response follows. It comes from nish/net/http1-client-codec |
HTTP1_HEAD |
the final response's status line and fields are in: status, reason, minor (the HTTP minor version), names and values (names in lowercase), header(name) (any case), contentLength (the body's length, or -1 when the response is not framed by one; it is always -1 for a response to HEAD, a 204, a 304 and a 2xx to CONNECT, which have no body, so read header("content-length") there) |
HTTP1_BODY |
some body bytes, de-chunked: body[bodyOff .. bodyOff + bodyLen), valid until the next call |
HTTP1_END |
the response is complete, trailers included |
HTTP1_UPGRADE |
a CONNECT was answered with 2xx, or a request expectUpgrade named with 101, and the connection is a tunnel: tunnelBytes() answers what was fed past the head |
HTTP1_CLOSED |
the connection is done: the server closed it between responses, or the response before was not kept alive (keepAlive() false) and nothing more can follow it |
HTTP1_ERROR |
the response is refused: error holds the code and errorText says why |
HTTP1_UPGRADE, HTTP1_CLOSED and HTTP1_ERROR are final: every later call
answers them again. What is fed after HTTP1_CLOSED or HTTP1_ERROR is
dropped; what is fed after HTTP1_UPGRADE is kept for tunnelBytes().
How a response ends
-
A response to
HEAD, an interim 1xx, a 204 and a 304 have no body, whatever their fields say. Callp.expect(true, false)before a response toHEAD. -
A 2xx to
CONNECTis a tunnel; callp.expect(false, true)first. It answersHTTP1_HEAD, thenHTTP1_ENDwith no body, thenHTTP1_UPGRADEon the next call;tunnelBytes()means something only afterHTTP1_UPGRADE, and answers what was fed past the head. Any other answer toCONNECThas a body like any response. -
Transfer-Encoding: chunkedis de-chunked;Content-Lengthis that many bytes. -
Anything else is read until the server closes. Call
p.finish()when the socket reports the close, thennext()again.
A close anywhere else, inside a status line, the fields or a body, is the
error HTTP1_E_TRUNCATED, so a body cut short is never taken for a whole one.
expect holds for one response and is forgotten after its HTTP1_END, so a
program sending a second request on the connection calls it again.
keepAlive() answers whether the connection may carry the next request once
HTTP1_END has come: an HTTP/1.1 response unless Connection lists close,
an HTTP/1.0 response only when it lists keep-alive, and never a response
read until close, a tunnel or an error. On a kept-alive connection, keep
feeding the same parser; the next response follows the last.
What the parser refuses
A response the parser cannot read in exactly one way is refused, never
repaired, since a client that repairs it differently from a proxy on the path
disagrees with it about where the next response starts. Each code is a
constant of nish/net/http1-client-codec:
error |
Code | Refused |
|---|---|---|
| 1 | HTTP1_E_STATUS_LINE |
a malformed status line, or a status outside 100 to 599 |
| 2 | HTTP1_E_VERSION |
an HTTP major version other than 1 |
| 3 | HTTP1_E_BARE_LF |
a line ended by LF without CR |
| 4 | HTTP1_E_FIELD |
a malformed field line: a folded line, whitespace before the colon, a control byte in a value |
| 5 | HTTP1_E_SWITCH |
a 101 to a request that did not ask to switch protocols (expectUpgrade, below) |
| 6 | HTTP1_E_FRAMING |
Transfer-Encoding together with Content-Length |
| 7 | HTTP1_E_CODING |
a transfer coding other than one chunked, last, or one in an HTTP/1.0 response |
| 8 | HTTP1_E_LENGTH |
a Content-Length that is not one run of digits, or two that differ |
| 9 | HTTP1_E_CHUNK |
a malformed chunk size, extension or chunk ending |
| 10 | HTTP1_E_TRUNCATED |
the connection closed before the response ended |
| 11 | HTTP1_E_STATUS_LINE_LIMIT |
a status line over maxStatusLine |
| 12 | HTTP1_E_HEADER_LIMIT |
a header section over maxHeaderBytes |
| 13 | HTTP1_E_COUNT_LIMIT |
more fields than maxHeaders, trailers included |
| 14 | HTTP1_E_BODY_LIMIT |
a body over maxBody, refused before a byte of it is read when its length is declared |
| 15 | HTTP1_E_TRAILER_LIMIT |
a trailer section over maxTrailerBytes |
| 16 | HTTP1_E_CHUNK_LINE_LIMIT |
a chunk-size line over HTTP1_MAX_CHUNK_LINE (4,096 bytes) |
| 17 | HTTP1_E_INFORMATIONAL_LIMIT |
more than HTTP1_MAX_INFORMATIONAL (32) interim responses to one request |
After an error the connection cannot be trusted: close it.
Switching protocols
A request that asks the server to switch protocols, such as a WebSocket's
opening handshake, is written with http1UpgradeRequestHead(method, target, names, values, upgrade), which adds Upgrade and Connection: Upgrade
itself. Tell the parser before you feed it the answer: call expectUpgrade()
after expect. A 101 then answers HTTP1_HEAD, HTTP1_END and
HTTP1_UPGRADE, and tunnelBytes() answers whatever was fed past its head,
which is the new protocol's, not HTTP's. Any other answer reads as it would
without the call. It lasts one request: the next expect clears it, and it
does nothing once a response has begun. WebSocket shows the
whole handshake.
Limits
new Http1ResponseParser(maxStatusLine, maxHeaderBytes, maxHeaders, maxBody, maxTrailerBytes) takes every limit a server could push against. maxBody is
an i64; the rest are i32. A negative limit is taken as 0. What the parser
holds is what you fed and it has not handed out yet, which buffered()
answers: feed what the socket gave you, then drain next(), and the parser
never holds more than a line in progress.
Each response's reason phrase and fields are kept as strings in the arena
until your program resets it, at most maxStatusLine + maxHeaderBytes for
each response.
Where next
- Networking opens the sockets these bytes travel over.
- The standard library lists every module.

