Networking
nish:net gives a program TCP and UDP sockets and a loop that waits for
whichever of them is ready. It is a thin layer: every socket is non-blocking,
there is no callback, and an async function cannot
wait on a socket yet, so your main is the loop. The protocol
stack on top of it (TLS, HTTP/1.1, HTTP/2, HTTP/3) is in the
standard library; this page is about the sockets
themselves, and Unix sockets covers the Unix-domain
listeners, connections and pairs, passing a descriptor over one, and naming
the process at the other end. The rules are in the reference,
nish:net.
Every function is also a global, as every nish: export is, but importing them
says where they come from. They are native only: a wasm32 build has no
sockets, and the compiler refuses a call to any of them there.
Errors are numbers
Every call answers an i32. Zero or more is success: a descriptor, a count of
bytes, or 0. A negative number is the failure's error number, negated, and
the ones a loop branches on are the same on every platform:
| Answer | Meaning |
|---|---|
-11 |
would block: nothing to read, no room to write, nobody waiting to be accepted. Call again once the loop says the socket is ready |
-32 |
the peer has gone. Writing to it never kills the process with SIGPIPE |
-98 |
the port is already in use |
-111 |
the connection was refused |
-22 |
a bad argument: a host that is not a numeric address, a flag that does not exist |
Hosts are numeric literals, "127.0.0.1" or "::1": the socket calls never
look a name up. To reach a name, resolve it first with
nish/net/dns. "::" listens on every address, IPv4 and
IPv6 both.
A server
tcpListen(host, port, backlog) opens a listening socket; port 0 lets the
kernel choose one, and netLocalPort(fd) tells you which. tcpAccept(fd, peer) takes the next waiting connection and writes its address into peer,
an 18-byte u8[]. netRead and netWrite move bytes between a socket and a
range of a u8[], and netClose closes it.
To wait, make a loop with pollCreate(), watch each socket with
pollAdd(loop, fd, events, token) — 1 for readable, 2 for writable, and a
token you choose to tell the sockets apart — and call
pollWait(loop, ready, timeoutMs). It fills ready with pairs, a token and
its events, and answers how many, 0 when the timeout passed.
import { netClose, netLocalPort, netRead, netWrite, pollAdd, pollCreate, pollWait, tcpAccept, tcpListen } from "nish:net";
export const main = (): i32 => {
const server = tcpListen("127.0.0.1", 0, 16);
if (server < 0) {
return 1;
}
console.log(`listening on ${netLocalPort(server)}`);
const loop = pollCreate();
pollAdd(loop, server, 1, 0);
const ready: i32[] = [0, 0];
const peer = new Array<u8>(18);
const buf = new Array<u8>(4096);
// Wait up to a second for one client, and echo back what it sends first.
if (pollWait(loop, ready, 1000) > 0) {
const conn = tcpAccept(server, peer);
if (conn >= 0) {
pollAdd(loop, conn, 1, 1);
if (pollWait(loop, ready, 1000) > 0) {
const n = netRead(conn, buf, 0, buf.length);
if (n > 0) {
netWrite(conn, buf, 0, n);
}
}
netClose(conn);
}
}
netClose(server);
netClose(loop);
return 0;
};
A real server keeps accepting until tcpAccept answers -11, keeps reading
until netRead does, and holds on to what netWrite could not send yet. The
loop is level-triggered: anything you leave unread is reported again on the
next pollWait.
The protocol servers in nish/net keep no timer of their own: your loop owns
the clock. Call their expire(monotonicNanos()) once a turn, with a
pollWait timeout no longer than the shortest deadline, and they close what
has gone on too long. Http1Server closes an idle slot, a head past
headTimeout, and a refused client still draining past lingerTimeout
(it shuts its write side and reads what the client still sends, up to
lingerBytes, so the error response is not lost to a reset).
Http2TlsServer closes a client that has not acknowledged its SETTINGS
within settingsTimeout, or is quiet past idleTimeout. Http3Server
needs no expire: its timer wheel already runs on the clock your loop hands
tick, so it closes a slot with no request open past
Http3Config.idleTimeout (60 s), and one whose client trickles a request's
head past headTimeout (30 s), by itself. A QUIC PING keeps the connection,
not the slot. A WebTransport layer ends a session quiet past its
idleTimeout and drops datagrams past maxDatagramsPerSecond; dropping is
the whole defence, as datagrams may be lost anyway, so a burst never closes
the session. Each limit has a default and is checked at start-up;
the standard library's nish/net section has
the details.
Keeping a request loop's memory flat
A server's loop runs until it is stopped, so whatever a pass allocates and
does not give back grows for as long as it serves. A pass that calls
accept or respond stores into the server's memory, and the compiler then
keeps everything the pass allocated until the loop ends: a reply text built
with a template on every request piles up. The compiler warns (NL9018) and
names the line. Build the temporary inside a block that opens with
using a = arena(), copying what must outlive it into a buffer you made
before the loop, and answer after the block:
while (event !== H1_NEED_MORE && event !== H1_ERROR) {
if (event === H1_END) {
{
using a = arena();
fill(heads, `request ${heads.served}\n`); // copies into heads.reply
}
answer(server.connection(slot), heads); // calls respond, outside the block
}
event = server.next(slot);
}
Over 20,000 requests the first version grew 1,252 kB and this one none. The
compiler refuses the block if anything allocated in it could outlive it, so
the rewrite cannot lose a value you still need. --deny-retention turns the
warning into an error, as it does for the
other retention warnings.
The same holds when the text is built in a handler the loop calls. A helper that builds the reply and also answers stores into the server's memory, so it has no scope of its own either, and what it drops piles up in the loop. The warning sits on the loop and names the helper's line. Put the block inside the helper, around the work that builds the text:
const readPost = (conn: Http1Connection, heads: Heads): void => {
{
using a = arena();
fill(heads, `request ${heads.served}\n`);
}
answer(conn, heads);
};
How many connections: the descriptor limit
Every connection a server holds is a file descriptor, and the operating system caps how many one process may have open with two limits: a soft one, which the process may raise itself, and a hard one above it, which only the operator may. Shells and service managers often start a program with a soft limit of 1024 (256 on macOS), far below a hard limit of 65536 or more.
A native program's first nish:net call that makes a descriptor
(tcpListen, tcpConnect, udpBind, unixListen, unixConnect,
unixSocketPair or pollCreate) raises the soft limit to the hard one, once
for the whole process, as Node and Go do at start-up. On macOS it stops at
OPEN_MAX (10240); where kern.maxfilesperproc is set lower, the raise
fails and the program keeps the limit it had. A program that never calls
nish:net keeps the limit it was given, and so does one that only accepts on
a listener it was handed (recvFd, socket activation) without making a
descriptor of its own.
Past the hard limit is the operator's to set: ulimit -Hn in the shell that
starts the server, LimitNOFILE= in a systemd unit, or the container's
nofile limit (docker run --ulimit nofile=…). At the limit, a call that
wants a descriptor answers -24 (EMFILE), and nish/net/handler waits for
one to close rather than spin (on -23, a full set of ring slots, too). On
Linux a listener can serve past the limit instead: see
More connections than descriptors.
A child the program starts with spawn inherits the raised soft limit, as
it would under Node. A child that relies on the lower one should lower it
itself.
Sharing a port and turning Nagle off: tcpListenFlags
tcpListenFlags(host, port, backlog, flags) is tcpListen with one more
argument, a set of flags added together. Flag 1 sets SO_REUSEPORT, which lets several listeners that all
ask for it open the same port, each with its own queue of waiting connections:
- On Linux the kernel spreads new connections across them. Start the same server several times, each process with its own listener on one port, and the connections are shared between the processes without a load balancer in front. Only listeners started by the same user may share.
- On macOS they share the port, but one listener takes the connections. It still lets a new copy of a server open the port before the old one closes, so a restart drops nothing.
Flag 2 sets TCP_NODELAY on every connection tcpAccept takes from the
listener, on every platform. Without it the kernel may hold a small write back
until the peer acknowledges the last one (Nagle's algorithm), which saves
packets for a stream of tiny writes and costs a round trip for a server that
answers each request with one small write. The bytes that arrive are the same
either way; only when they leave changes.
A socket tcpConnect opens holds TCP_NODELAY already, as libpq and Go set
it on every client socket: a client sends a whole request and waits on the
answer, so there is nothing for Nagle to save.
Flag 0 is tcpListen exactly and 3 is both. Flag 4 steers connections
for nish/net/prefork, and flag
8 is the next section's. Any other bit answers -22.
A listener without flag 1 cannot open a port a listener with it holds, nor
the other way round: both answer -98.
import { netClose, netLocalPort, tcpListenFlags } from "nish:net";
export const main = (): i32 => {
const first = tcpListenFlags("127.0.0.1", 0, 64, 1);
if (first < 0) {
return 1;
}
const port = netLocalPort(first);
// The same port again: allowed, because both listeners asked for it.
const second = tcpListenFlags("127.0.0.1", port, 64, 1);
console.log(`${second >= 0}`); // true
console.log(`${tcpListenFlags("127.0.0.1", port, 64, 4)}`); // -22
netClose(second);
netClose(first);
return 0;
};
A program can watch both listeners in one loop, as in the server above, and
accept from whichever is ready. A server that wants both the shared port and
its small answers sent at once opens each listener with 3.
More connections than descriptors: NET_LISTEN_RING
On Linux a listener opened with the flag NET_LISTEN_RING (8) takes its
connections into io_uring rings instead of the descriptor table. What
tcpAccept answers is then a slot handle, a number from 1048576 up,
and the descriptor limit no longer bounds how many a process holds: one
listener serves thousands of connections under a nofile limit of 1024,
with only a few dozen descriptors open.
A slot handle goes wherever a server loop puts a connection. netRead,
netWrite, netWrite2, netClose, netLocalPort and the poll calls take
it as they take a descriptor, with the same answers: -11 until pollWait
reports the handle ready, then progress, and 0 at the end of the stream. A
loop written for descriptors runs unchanged. What differs:
-
netShutdownqueues the shutdown behind the bytes already written, so the peer reads them and then the end of the stream. After the write side is shut,netWriteanswers-32; after the read side is,netReadanswers what is queued and then0.Http1Server's lingering close works on a slot as on a descriptor. -
netMssanswers-1: the ring cannot ask the socket for its segment size, so take your fallback. -
netSetOptionanswers-95, and every call that needs a real descriptor (sendFd,peerCredentials) answers-9, as does any call on a handle after itsnetClose. tcpAcceptwrites the peer's address as zeros.-
A slot handle is in one poll loop at a time:
pollAddto a second loop answers-17until the first removes it or is closed. -
Once a connection has failed, by a peer's reset or a failed send,
netWriteanswers the errno (-104for a reset), andnetReadanswers it on every read once the queued bytes are read, where a descriptor answers it once and then0. -
When every slot the listener may hold is taken,
tcpAcceptanswers-23. Each ring listener holds up to 64 rings of as many slots as the descriptor limit, at most 16,384 a ring. Close a connection and the next waiting one takes its slot. -
If the kernel is short of memory for a new connection,
tcpAcceptanswers-12or-105once, as it does for a plain listener. The listener then pauses for a moment (10 ms, doubling to 640 ms while the shortage lasts) and goes on accepting; waiting connections stay queued. -
If the listener's accept fails with any other error but
-11,-4,-103or-71, the listener is broken: it stops accepting andtcpAcceptanswers that error from then on. Close the listener and listen again. -
A process opens at most 31 ring listeners in its life, closed ones
included; the 32nd
tcpListenFlagswithNET_LISTEN_RINGanswers-23. -
Each connection holds at most 256 KiB that the peer has not taken yet. A
netWritepast that copies what fits; a full connection answers-11until the peer reads, as a full socket buffer does.
The kernel may refuse rings: a seccomp filter (some container runtimes), a
kernel without io_uring, or io_uring_disabled set. Then the listener
serves descriptors as if the flag were not there, and netRingState() says
what happened: 1 for rings, 0 until a listener has found rings or a
refusal, or the refusal as a negative errno (-1 when a filter or
io_uring_disabled forbids it, -38 for a kernel without io_uring). The
first listener that finds rings or a refusal decides for the whole process.
A listener whose rings fail for want of descriptors or memory serves
descriptors itself, and a later listener asks again.
On macOS the flag is accepted and does nothing.
import { NET_LISTEN_RING, netClose, netRingState, tcpListenFlags } from "nish:net";
export const main = (): i32 => {
const listener = tcpListenFlags("127.0.0.1", 0, 1024, 1 | NET_LISTEN_RING);
if (listener < 0) {
return 1;
}
const state = netRingState();
if (state === 1) {
console.log("connections go into rings");
} else {
console.log(`descriptors, because rings answered ${state}`);
}
netClose(listener);
return 0;
};
Use the flag when a server needs more connections than its limit allows and
nothing it does with a connection needs a descriptor. A server that hands
connections to another process with sendFd keeps descriptors.
nish/net/handler takes the flag through its limits. Set
HandlerLimits.ring and open the listener with limits.listen(host, port, backlog, flags), which is tcpListenFlags with NET_LISTEN_RING added when
ring is set, then hand it to new HttpHandler(limits, listener, router) as
any listener. HandlerLimits.slots is then the cap on connections, not the
descriptor limit: set it to what the server should hold. When every ring slot
is taken, the handler rests its listener on the -23, as it does on -24
for descriptors, until a connection closes or 100 ms pass, so a full server
waits rather than spin. h.resting() is true while it does, for a server
that wants to log or count the times it is full:
import { HandlerLimits, HttpHandler } from "nish/net/handler";
import { Router, RouterLimits } from "nish/net/router";
import { netClose, netRingState } from "nish:net";
export const main = (): i32 => {
const limits = new HandlerLimits();
limits.ring = true;
limits.slots = 100000;
const listener = limits.listen("127.0.0.1", 0, 4096, 0);
if (listener < 0) {
return 1;
}
const h = new HttpHandler(limits, listener, new Router(new RouterLimits()));
console.log(netRingState() === 1 ? "connections go into rings" : "descriptors");
h.drain();
netClose(listener);
return 0;
};
Rings are off by default. What one ring process costs against five processes on descriptors, with an SQLite lookup per request, is measured in the benchmarks.
Socket options: netSetOption
netSetOption(fd, option, value) sets one option on a socket. The option is
one of five ids nish:net exports, and any other number answers -22:
| Option | value |
|---|---|
NET_TCP_NODELAY |
1 sends small writes at once, 0 lets Nagle batch them |
NET_TCP_QUICKACK |
1 acknowledges what arrives at once, 0 delays the ACK; Linux only, -95 on macOS |
NET_SO_KEEPALIVE |
1 probes a connection that has gone quiet |
NET_SO_SNDBUF |
the send buffer, in bytes (Linux stores twice the value) |
NET_SO_RCVBUF |
the receive buffer, in bytes (Linux stores twice the value) |
It answers 0, or a negative error: -9 for a closed descriptor, -88 for
one that is not a socket.
On Linux a keep-alive connection already delays its ACKs: once the server
has sent its first response, the kernel waits to acknowledge the next request
until it can ride on that request's response. Most servers need no call at
all. Setting NET_TCP_QUICKACK changes that, and the trade runs both ways:
-
1after each response turns delayed ACKs off until the kernel next turns them on, so every request is acknowledged at once in a segment of its own. That is one more system call and one more packet per request, worth it only for a peer that waits on the ACK before it sends more. -
0once, after accept, keeps delayed ACKs on from the first request. It costs one call per connection. The kernel still switches them off for the next request when a response took longer than its ACK timer (tens of milliseconds) or a segment arrived out of order, and nothing short of a call after each response turns them back on that early. On a connection answered quickly, that call after each response changes nothing. - No call behaves like
0once from the second request on.
bench/quickack/ in the repository measures each way. On four cores
running Linux, at 256 keep-alive connections over loopback, 1 after
each response added one system call and one segment per request and served
13 to 18% fewer requests per second than no call. 0 after each response and
0 once stayed within the noise, since 0 saves an ACK only after a slow
response. A server that wants the delayed ACKs from the
first request sets the option once, as it accepts:
import { NET_TCP_QUICKACK, netSetOption, tcpAccept } from "nish:net";
export const acceptOne = (listener: i32, peer: u8[]): i32 => {
const conn = tcpAccept(listener, peer);
if (conn >= 0) {
// Delay ACKs from the first request, so each one rides on its response.
// On macOS this answers -95, which a server can ignore.
netSetOption(conn, NET_TCP_QUICKACK, 0);
}
return conn;
};
The ids are plain i32s you pass straight to the call. They are not module
constants, so const MODE = NET_TCP_QUICKACK; at the top of a module is an
error; write the id where you call netSetOption.
A connection's segment size: netMss
netMss(fd) answers how many bytes of payload one TCP segment on a connected
socket carries, the maximum segment size the kernel is sending with now. A
program that batches writes can size each batch to whole segments. It
answers -1 when there is no MSS to read: a socket that is not connected yet,
a listener, a UDP socket, a closed descriptor or one that is not a socket,
and a NET_LISTEN_RING slot handle.
-1 is not an error code, so check for a positive answer and fall back to a
size of your own:
import { netMss } from "nish:net";
// Bytes to gather before one write: four full segments, or 4096 when the
// socket has no MSS to tell.
export const batchSize = (conn: i32): i32 => {
const mss = netMss(conn);
return mss > 0 ? mss * 4 : 4096;
};
The kernel can change the MSS during a connection, as it learns the path, so
read it again rather than keeping the first answer. On Linux it comes from
TCP_INFO, on macOS from TCP_MAXSEG; over loopback it is large (tens of
kilobytes), and over Ethernet about 1,448. A wasm build, which has no sockets,
compiles netMss to -1, so the same code builds there and takes the
fallback.
A server on every core: nish/net/prefork
A Nish server is one thread. To use every core, run one copy of it per core,
each with its own SO_REUSEPORT listener on the same port: that is what
nish/net/prefork does. Call preforkWorkers(process.argv) at the top of
main, before you open a listener:
import { netClose, tcpListenFlags } from "nish:net";
import { PREFORK_STOPPED, preforkWorkers } from "nish/net/prefork";
export const main = (): i32 => {
const worker = preforkWorkers(process.argv);
if (worker < 0) {
// The process you started: every worker has exited.
return worker === PREFORK_STOPPED ? 0 : 1;
}
// A worker, numbered from 0: open its own listener and serve.
const listener = tcpListenFlags("0.0.0.0", 8080, 1024, 1);
if (listener < 0) {
return 1;
}
// ... the readiness loop of a one-process server, unchanged ...
netClose(listener);
return 0;
};
The program runs in two roles. The process you start becomes the supervisor:
it starts the program again once per worker, and preforkWorkers answers it
only when they have all exited, with -1 (PREFORK_STOPPED) after a shutdown
or -2 (PREFORK_SPAWN_FAILED) when a worker's first start failed or the
supervisor could not set up its signal handling, in which case no worker
starts.
In each worker it answers that worker's index, 0 up to one less than the
count, at once. A worker is told its index by one extra argument at the end of
its arguments; preforkArgs(process.argv) gives you the arguments without it,
the same list in every process, so read your port and options from that.
Workers share nothing, so each one is the single-threaded server it was before.
-
How many workers. The
NISH_WORKERSenvironment variable when it is a positive whole number, such asNISH_WORKERS=2; anything else in it is ignored. Otherwise one per core on Linux, and one on macOS, which lets the listeners share the port but does not spread connections across them. The count has no upper bound, so setNISH_WORKERSyourself; never pass it on from a client. - Restarts. A worker that exits, whatever its exit code, or is killed, is started again at the same index. One that dies within a second of starting waits first: 100 ms, then twice as long each time it does so again, up to 5 s. A server that crashes on start costs one start every few seconds, not a core. A restart the system refuses (out of processes or memory) waits the same way and is tried again.
-
Stopping. SIGTERM or SIGINT (Ctrl-C) to the supervisor sends SIGTERM to
every worker and waits up to 5 s for them, then kills the rest; a second
signal kills them at once. The supervisor then exits with the
-1path above. A supervisor stopped any other way (SIGKILL, the out-of-memory killer, SIGHUP or SIGQUIT) leaves its workers running; stop it with SIGTERM.
preforkWith(process.argv, config) takes a PreforkConfig to change those
numbers: workers, backoffMs, backoffCapMs, quickExitMs (what counts as
dying young), graceMs and tickMs (how often the supervisor looks for a
restart that is due). A negative number, a tickMs or backoffMs of zero,
or a backoffCapMs below backoffMs stops the program with a panic naming
the field. examples/prefork-server.ts is a whole HTTP server built this way:
each answer names the worker that sent it, so a few curls show the spread.
By default the workers are not pinned to cores. Each one binds the port with
SO_REUSEPORT, the kernel picks a worker for each new connection by hashing
its address, and the scheduler runs each worker on any core it allows. To keep
each worker on its own core and steer each connection to it, use a
config-owned listener with pin, below.
One core per worker
Set listenPort and the supervisor binds the listeners itself, one per
worker, before any worker starts: worker k takes its own with
preforkListener(process.argv) and accepts from it, and gets the same one
back after a restart. listenHost, listenBacklog and listenFlags (0, or
2 for TCP_NODELAY) shape them; port 0 picks a free port for the whole
group, and with listenUdp one that no UDP or TCP socket holds, so two
groups on port 0 never share a port and never receive each other's
datagrams.
Add pin = true and worker k also pins itself to CPU k
(setCpuAffinity), and on Linux the listeners steer each connection to the
worker on the CPU that received it, so a connection's packets and the worker
that answers it share a core's caches:
import { PREFORK_STOPPED, PreforkConfig, preforkListener, preforkWith } from "nish/net/prefork";
export const main = (): i32 => {
const config = new PreforkConfig();
config.listenPort = 8080;
config.pin = true;
const worker = preforkWith(process.argv, config);
if (worker === PREFORK_STOPPED) {
return 0;
}
if (worker < 0) {
return 1;
}
const listener = preforkListener(process.argv);
// Accept and serve on `listener` here.
return listener >= 0 ? 0 : 1;
};
Pinning is for a machine the server has to itself: each worker owns a core,
so workers should be at most availableParallelism(), and on Linux more pinned
workers than CPUs stops the program with a panic rather than doubling two
workers up on one core. That count is availableParallelism(), which a
container's CPU quota or NISH_NUM_THREADS lowers.
A CPU outside the ones the program may run on (taskset, a container's cpuset)
is never taken, and worker k asks for CPU number k, not the k-th CPU
allowed: under taskset -c 4-7 every pin fails, each worker says so on stderr
and runs unpinned, and a connection received on CPU 4 or above, past the
listeners, falls back to the kernel's hash. Run a pinned server on CPUs that
start at 0. Off Linux the workers run unpinned and unsteered, and the
supervisor says so on stderr. A listenPort outside -1..65535, a
listenBacklog of 0 or less, or other listenFlags stop the program with a
panic.
QUIC on every core: listenUdp
A QUIC connection is named by its connection ID, not by the client's address,
so it survives the client moving to a new port or network. Over several
workers that only holds if each datagram reaches the worker that holds its
connection. Set listenUdp with listenPort and the supervisor binds one UDP
socket per worker on that port, as a group the kernel steers by connection ID:
a datagram whose destination connection ID starts with byte k goes to worker
k. Worker k takes its socket with preforkUdpListener(process.argv) and
the group's size with preforkUdpCount(process.argv), and puts both in its
QuicServerConfig, so every connection ID it hands out starts with k:
import { PREFORK_STOPPED, PreforkConfig, preforkUdpCount, preforkUdpListener, preforkWith } from "nish/net/prefork";
import { QuicServerConfig } from "nish/net/quic";
/** Worker `worker`'s QUIC settings: every connection ID it issues starts with its index. */
export const steered = (config: QuicServerConfig, worker: i32): QuicServerConfig => {
config.workerIndex = worker;
config.workerCount = preforkUdpCount(process.argv);
return config;
};
export const main = (): i32 => {
const config = new PreforkConfig();
config.listenPort = 4433;
config.listenUdp = true;
const worker = preforkWith(process.argv, config);
if (worker === PREFORK_STOPPED) {
return 0;
}
if (worker < 0) {
return 1;
}
const udp = preforkUdpListener(process.argv);
// Serve HTTP/3 here: `new Http3Server(steered(quicConfig, worker), h3Config, udp, slots, entropy)`.
return udp >= 0 ? 0 : 1;
};
A server that is not part of a group leaves workerIndex at -1 and
workerCount at 0. Any other pair outside 0 <= workerIndex < workerCount <= 255 stops the program with a panic when the QuicListener or
Http3Server is built, naming the pair:
nish/net/quic: workerIndex 2 and workerCount 2, not 0 <= k < n <= 255
Two more things change in a group. A worker that receives a packet for a connection ID that another worker issued drops it instead of answering with a stateless reset, since that reset would end the other worker's live connection. And the group holds at most 255 workers, as the steering reads one byte: more stops the program with a panic.
Steering needs Linux. Where the kernel refuses it (off Linux, or under a
seccomp profile that forbids the program), the supervisor says so on stderr,
nish/net/prefork: QUIC steering refused (-95), running one worker, and runs
a single worker on a plain socket, with preforkUdpCount answering 1.
A serviceWorker other than 0 has no place in that one-worker run, so
there the supervisor starts nothing and answers PREFORK_SPAWN_FAILED.
The underlying call is udpBind with flag 4
(nish:net).
QUIC congestion control: congestion
QuicServerConfig.congestion picks the congestion controller every
connection runs: QUIC_RECOVERY_NEWRENO (0, RFC 9002) or
QUIC_RECOVERY_CUBIC (1, RFC 9438), both from nish/net/quic-recovery.
The field is required; pass QUIC_RECOVERY_DEFAULT, which names CUBIC,
unless you have a reason to pick. Any other value is refused: a connection
built with it closes at once, as one with any other limit out of range does.
CUBIC regrows the window faster after a loss on a long, lossy path, and on a
transfer of one bandwidth-delay product the two finish level. The
measurement is in
the QUIC security record.
QUIC and ECN: arrivalEcn
Http3Server uses Explicit Congestion Notification (ECN) in both
directions without any setup. It marks its packets ECT(0) wherever the
platform sends the ECN bits, and halves its window when the client reports
congestion (CE). Its ACKs also report the bits each client packet arrived
with: from the first marked packet on, the ACKs of that packet-number space
are ACK_ECN frames, with ECT(0), ECT(1) and CE counts for the space (RFC 9000
§13.4.1). A client that validates its own path needs those counts, or it
stops marking.
A program that drives QuicListener itself reads the arrival bits from
udpRecvFrom's meta[1] and sets them on the connection before it hands
the datagram over:
conn.arrivalEcn = meta[1] & 3;
quicListenerReceiveSegments(conn, buf, 0, n, segment, now);
If arrivalEcn stays 0, which means Not-ECT, the connection counts
nothing and its ACKs stay plain. That is always the case on macOS, where
meta[1] is always 0. A packet delivered twice is counted once.
Workers that are not restarted
Set restart to false when something else restarts the service, such as
systemd or a container runtime, and a worker that exits should stay
down rather than come back on its own. It then stays gone,
and the others carry on. Once every worker has exited, the supervisor returns
-3 (PREFORK_WORKERS_EXITED), so the program can exit and let its own
supervisor decide:
import { PREFORK_STOPPED, PreforkConfig, preforkWith } from "nish/net/prefork";
export const main = (): number => {
const config = new PreforkConfig();
config.restart = false;
const worker = preforkWith(process.argv, config);
if (worker < 0) {
return worker === PREFORK_STOPPED ? 0 : 1;
}
// ... serve as worker `worker` ...
return 0;
};
SIGTERM and SIGINT still stop the workers that are left, and still give -1.
Channels to the supervisor and to a service worker
Workers share nothing, but some servers need one thing in one place: a cache,
a connection pool, a counter. Two more PreforkConfig fields give the
processes a Unix socketpair to talk over
(Unix sockets):
-
supervisorChannels,true: each worker gets a pair to the supervisor. Supervise withpreforkSupervise(process.argv, config, state, onChannel)instead ofpreforkWithto hear from them;preforkWithreads what a worker sends and drops it. -
serviceWorker, a worker's index: every other worker gets a pair to that one.-1, the default, gives none. -
channelKind:NET_UNIX_STREAM(1, the default), orNET_UNIX_SEQPACKET(2) to keep each write one message. Only Linux has seqpacket pairs, so stream is the default: it works on macOS too. On macOS a seqpacket channel stops the program with a panic,seqpacket channels need Linux, before any worker starts.
A worker finds its ends with three calls, each taking process.argv and
answering a descriptor, or -1 when it has no such end:
| Call | Answers |
|---|---|
preforkSupervisorChannel(process.argv) |
its end to the supervisor |
preforkServiceChannel(process.argv) |
its end to the service worker |
preforkServiceChannelOf(process.argv, index) |
in the service worker, its end to worker index |
The ends start at descriptor 3, the supervisor channel first. A worker holds
those ends and no other descriptor of the supervisor's. Like every nish:net socket they
never block, so wait for one in the readiness loop alongside the listener.
Where they are rides in one more argument, --nish-prefork-channels=, just
before the worker's index; preforkArgs drops it with the index.
import { NET_UNIX_STREAM, netRead } from "nish:net";
import {
PREFORK_STOPPED,
PreforkConfig,
preforkServiceChannel,
preforkServiceChannelOf,
preforkSupervise,
} from "nish/net/prefork";
class Log {
lines: i32 = 0;
}
// The supervisor: worker `index` wrote on `fd`, or hung up (`events & 4`).
const heard = (log: Log, index: i32, fd: i32, events: i32): void => {
const buf = new Array<u8>(256);
while (netRead(fd, buf, 0, buf.length) > 0) {
log.lines = log.lines + 1;
}
};
export const main = (): number => {
const config = new PreforkConfig();
config.supervisorChannels = true;
config.serviceWorker = 0;
config.channelKind = NET_UNIX_STREAM;
const worker = preforkSupervise(process.argv, config, new Log(), heard);
if (worker < 0) {
return worker === PREFORK_STOPPED ? 0 : 1;
}
if (worker === 0) {
// The service worker: its end to worker 1, say.
const one = preforkServiceChannelOf(process.argv, 1);
// ... serve the others over their ends ...
console.log(`worker 1 is on descriptor ${one}`);
} else {
const service = preforkServiceChannel(process.argv);
// ... ask the service worker over `service`, serve clients ...
console.log(`the service worker is on descriptor ${service}`);
}
return 0;
};
The callback runs in the supervisor each time a worker's end is readable or
has hung up, with events as pollWait gives it: 1 readable, 4 added on
a hang-up. Read everything waiting, since an end left readable calls it again
at once; write back if you like; never close the end. The supervisor closes
it when the worker hangs up or is reaped, and a restarted worker comes with a
new one. The state is whatever you pass, a class instance to keep counts in,
since a function cannot capture.
A restart restarts the group. A socketpair cannot be handed to a process
that is already running, so with a service worker the workers live and die
together: when any of them exits, the supervisor sends the rest SIGTERM
(SIGKILL after graceMs), and once none is left starts them all again on
fresh pairs, after the backoff of the one that exited first. No worker is left
holding an end to a peer that is gone. With only supervisor channels, a
worker restarts on its own with a new pair. With restart off nothing is
signalled: a worker sees its peer's exit as a hang-up on its end.
A serviceWorker below -1 or not below the worker count, more than 64
workers with a service worker (the most descriptors one child is handed: 63
with listenPort, 62 with listenUdp too), listenUdp with a negative
listenPort, a channelKind that is neither kind, or seqpacket channels off
Linux stop the program with a panic.
Serving a file
A file bigger than you want in memory goes out the way it is read: a window
at a time. Open it once with fileOpen, answer with fileSize as the
length, and move each window from fileReadAt into the response with
Http1Connection.write. Each connection keeps only an i64, how far into the
file it has got, and every connection shares one window, so a server that
sends a 1 GiB file holds what it holds for a 64 KiB one.
The output is the bound. write takes as much of the window as there is room
for and answers how much, or H1_AGAIN when there is none; stop there, keep
the position, and go on from it when next(slot) answers H1_WRITE, which
it does once half the output is free. A window taken in part is read again
from the new position:
const pump = (conn: Http1Connection, file: i32, size: i64, window: u8[], at: i64): i64 => {
let pos: i64 = at
while (pos < size) {
const left: i64 = size - pos
const want: i32 = left < toI64(window.length) ? toI32(left) : toI32(window.length)
const n: i32 = fileReadAt(file, window, toI32(0), want, pos)
if (n <= 0) {
conn.abort() // the file shrank or failed: the length can no longer be met
return toI64(-1)
}
const took: i32 = conn.write(window, toI32(0), n)
if (took === H1_AGAIN) {
return pos // the output is full: wait for H1_WRITE
}
if (took < 0) {
conn.abort()
return toI64(-1)
}
pos = pos + toI64(took)
if (took < n) {
return pos
}
}
conn.end()
return toI64(-1) // done
}
This pump moves the body only, and answers -1 alike when the body is sent
and when it aborted, since either way the connection owes nothing more. The
example also keeps a position for a response head that found no room in the
output and retries respond at the next H1_WRITE.
respond takes the body's length as an i32, so one response carries at most
2^31 - 1 bytes; refuse a larger file before you listen. A read that comes back
short of the length you announced means the file changed under you, and
abort() closes the connection rather than leave the client waiting for bytes
that will not come. A HEAD request (conn.parser.methodIs("HEAD")) needs
the head and no reads at all.
examples/file-server.ts
is the whole server: file-server big.mp4 8080, then
curl -s http://127.0.0.1:8080/ | cmp - big.mp4. Serving a 64 MiB file to a
slow reader, its peak memory is the same as serving a 64 KiB one.
Names: nish/net/dns
A socket call takes an address. nish/net/dns turns a name into addresses
without leaving your loop: a DnsClient joins the loop you made with
pollCreate, under a range of tokens of its own, sends its queries on sockets
it opens, and answers each lookup with the 18-byte addresses tcpConnect and
udpSendTo take, the port you gave already in them. Your loop hands it the
tokens it owns and the time; it never waits and never reads the clock itself,
so a slow lookup costs your other sockets nothing.
import { DnsClient } from "nish/net/dns";
import { DNS_RESOLVE_PENDING, DnsResolver } from "nish/net/dns-resolver";
import { DnsConfig, dnsParseResolvConf } from "nish/net/dns-config";
import { DNS_ADDRESS_BYTES } from "nish/net/dns-message";
import { pollCreate, pollWait, tcpConnect } from "nish:net";
import { monotonicNanos } from "nish:process";
export const main = (): i32 => {
// No file read here: an empty resolv.conf means the servers on 127.0.0.1 and ::1.
const config = new DnsConfig();
dnsParseResolvConf(new Array<u8>(0), 0, 0, config);
const loop: i32 = pollCreate();
const dns = new DnsClient(new DnsResolver(config, null, 4), loop, 1000);
const lookup: i32 = dns.start("example.com", 443, monotonicNanos());
if (lookup < 0) {
console.log(`not started: ${lookup}`);
return 1;
}
const out: u8[] = new Array<u8>(16 * DNS_ADDRESS_BYTES);
const ttls: i32[] = new Array<i32>(16);
const ready: i32[] = new Array<i32>(16);
let found: i32 = dns.result(lookup, out, ttls);
while (found === DNS_RESOLVE_PENDING) {
const count: i32 = pollWait(loop, ready, dns.timeoutMs(monotonicNanos(), -1));
const now: i64 = monotonicNanos();
for (let k: i32 = 0; k < count; k++) {
if (dns.owns(ready[2 * k])) {
dns.ready(ready[2 * k], ready[2 * k + 1], now);
}
}
dns.tick(now);
found = dns.result(lookup, out, ttls);
}
if (found <= 0) {
console.log(`no address: ${found}`);
return 1;
}
// The first address, port 443 already in it, is what tcpConnect takes.
const addr: u8[] = new Array<u8>(DNS_ADDRESS_BYTES);
for (let i: i32 = 0; i < DNS_ADDRESS_BYTES; i++) {
addr[i] = out[i];
}
const fd: i32 = tcpConnect(addr);
console.log(`${found} addresses, connecting: ${fd >= 0}`);
return 0;
};
start answers a lookup number, or a negative code when it cannot start one:
-44 every slot is in use, and -1 to -6 (DNS_ERR_* of
nish/net/dns-message) the name is not one, such as "" or a label over 63
bytes. Check it before you pass it to result, which panics on a number that is
not a lookup.
A lookup answers its address count, A records first, or a negative code:
-41 the name has no address (it does not exist, or exists with no A or AAAA
record), -42 no server answered, -43 an alias chain ran past eight aliases
or led back to the name asked (a loop inside one reply reads as -41), and from
resolveHost -45 its time ran out. Another negative number is the errno of a
socket or of pollWait that failed.
resolveHost answers start's codes too, and waits for both the A and the
AAAA answer until the time runs out; then a family that answered with
addresses is returned on its own, and only a lookup with none yet is -45.
localhost, an address literal and a name in the hosts table answer at once,
with no packet. For a command-line tool or a program's
start-up, resolveHost(name, timeoutMs, port, out, ttls) reads
/etc/resolv.conf and /etc/hosts and blocks until the lookup ends. The
standard library page has both in full.
TLS to a server: tlsConnect
nish/net/tls-tcp's tlsConnect(addr, serverName, alpn, timeoutMs) takes an
address like tcpConnect's, connects, runs a TLS 1.3 handshake and checks
the server's certificate for serverName, blocking until it has or the time
has run out. Like the socket calls it never looks a name up: resolve one
first, as above, and pass the name you meant as serverName. The client's
writeAll and readWait then write and read under a timeout, for a program
with no loop of its own. Their answers, and how to drive the client from
your own loop instead, are on
TLS client, with how to trust roots
other than the system's.
HTTP/2 without TLS: nish/net/http2-cleartext
Cleartext has no confidentiality and no integrity. Anyone on the path reads every request and response, cookies included, and can change them. Serve h2c only behind a proxy that terminates TLS, on loopback, or on a private network. For an open network use
nish/net/http2-tls.
Http2CleartextServer serves HTTP/2 over plain TCP to a client that already
knows the port speaks it ("prior knowledge", RFC 9113 §3.3): the client's
first bytes are HTTP/2's 24-byte connection preface. There is no
Upgrade: h2c, which RFC 9113 deprecates, and no HTTP/1.1 on the same port.
It is nish/net/http2-tls's loop with TCP in place of TLS: the same
accept, readable, writable, next, flush, close and rest, and
the same interest bits (read 1, write 2, done 16).
import { pollAdd, pollModify, tcpListen } from "nish:net";
import { H2_ERROR, H2_NEED_MORE, H2_REQUEST, Http2Config, Http2Connection } from "nish/net/http2";
import { H2C_DONE, Http2CleartextServer } from "nish/net/http2-cleartext";
const LISTENER: i32 = 64; // past every slot's token
const start = (loop: i32): Http2CleartextServer => {
const listener: i32 = tcpListen("127.0.0.1", 8080, 128);
pollAdd(loop, listener, 1, LISTENER);
// 64 slots; a client has 10 seconds to send its preface.
return new Http2CleartextServer(new Http2Config(), listener, 64, 10000);
};
const serve = (loop: i32, server: Http2CleartextServer, slot: i32, events: i32): void => {
let wants: i32 = (events & 5) !== 0 ? server.readable(slot) : server.writable(slot);
const none: u8[][] = [];
let event: i32 = server.next(slot);
while (event !== H2_NEED_MORE && event !== H2_ERROR) {
const conn: Http2Connection = server.connection(slot);
if (event === H2_REQUEST) {
conn.respond(conn.stream, 200, none, none, true);
}
event = server.next(slot);
}
wants = server.flush(slot);
if ((wants & H2C_DONE) !== 0) {
server.close(slot);
} else {
pollModify(loop, server.fd(slot), wants & 3, slot);
}
};
When the listener's token comes back, server.accept() answers the new
slot, which you pollAdd under its own number as the token, or -105
(H2C_POOL_FULL) when every slot is busy, having closed that connection.
A client that sends anything but the preface, an HTTP/1.1 request line
included, is closed at the first byte that differs and is sent nothing.
One that sends part of it and stops holds its slot until you call
server.expire(monotonicNanos()) past the timeout the constructor took;
call it every so often, say whenever pollWait returns. Once the preface is
in, the slot has no clock of its own: close one you judge idle yourself.
curl --http2-prior-knowledge http://127.0.0.1:8080/ talks to it. What it
refuses and why is its security record.
Trusting a peer's address
A TCP peer's address is proven: the handshake that tcpAccept completes needs
answers sent to it. A UDP datagram's source address is only what its sender
wrote, so anyone can send one that names another host. A server that caps or
allows clients by address keys that on a UDP address only once a handshake has
proven it.
For QUIC, Http3Server.addresses[slot] is the source of the slot's
first datagram, and it is proven once server.quic(slot).addressValidated is
set: by a Handshake packet from the client, or by a Retry token when the
server's config turns retry on. The relay example in examples/relay counts
a client against its per-address cap only from that point.
Under Node
A program that uses nish:net type-checks with npm run check, but it has no
reading under Node: Node's sockets become ready only to Node's own event loop,
which a program that owns its loop never returns to, so each call throws there,
netMss included. Run it natively.

