Security record: QUIC on every core — steering, worker-marked IDs and the UDP group
The record of QUIC on more than one core, the design in
docs/h3-multicore.md: a prefork group of N workers on
one UDP port, where the kernel hands each datagram to the worker whose index
is byte 0 of its destination connection ID. It covers the three places that
make that work: udpBind flag 4 in the runtime (the classic-BPF reuseport
program), the worker byte in every connection ID a QuicListener and a
QuicConnection issue, and the supervisor in nish/net/prefork that binds
the group and hands socket k to worker k.
It says what each refuses and with what, where a cap is set, what a peer can
choose, and the test that pins each property. It is written with the code
rather than after an audit, at fb2a4d54 on the stage's branch (PR #1470); function
names are given rather than line numbers. QUIC itself is
quic.md, the HTTP/3 carrier http3.md, and the
runtime's sockets runtime.md (RT-17 is amended for flag 4).
Result. A datagram reaches the worker its connection ID names, for a short header and a long one, and a worker never answers a datagram a sibling owns with a stateless reset, which would end that sibling's live connection. A kernel that refuses the steering program leaves one worker serving the port, never N workers that would misroute each other's packets. Two findings are closed here (QM-1, QM-2) and three are accepted, Low (QM-3 to QM-5): a peer picks the worker its first Initial lands on, byte 0 of every ID tells an observer which worker issued it, and a process of the same user can join the group and replace its program.
Scope
| File | What was read |
|---|---|
the UDP path of nish/net/prefork |
preforkBindUdp (the group, its fallback and its failure paths), the --nish-prefork-udp= flag (preforkUdpListener, preforkUdpCount, preforkUdpField), what preforkArgs strips of it, the worker count's cap with listenUdp, and how a restarted worker gets the same socket. Not the rest of the supervisor: its channels, the RT-40 descriptor handoff, pin and restarts stay in the index's "Not yet audited" row |
std/net/quic.ts |
QuicServerConfig.workerIndex and workerCount, quicWorkerCheck and its panic, quicWorkerByte, and where the byte is written into the first SCID and every NEW_CONNECTION_ID |
std/net/quic-listener.ts |
the no-reset rule in handleWindow, and the worker byte in the Retry SCID |
std/net/quic-conn-cid.ts |
that an ID issued later keeps byte 0 and draws the rest as before |
Read beside them, without owning them: nish_udp_bind in
runtime/runtime-net.c (flag 4, recorded in RT-17 of
runtime.md), std/net/http3-server.ts (which passes the
configuration through) and tests/runtime-test.c (test_udp_steer).
Threat model
The attacker is any host that can send UDP datagrams to the port, and an
on-path observer of them. It chooses every byte of every datagram it sends,
including the destination connection ID of a first Initial and of a short
header packet it forges; it does not hold the server's reset-token key or
any connection's keys. Whoever starts the program chooses its arguments and
environment (NISH_WORKERS, the --nish-prefork-* flags), as for the rest
of nish/net/prefork: they are not attacker input.
Findings
| ID | Severity | Where | Finding | Status |
|---|---|---|---|---|
| QM-1 | Low | std/net/quic-listener.ts (handleWindow) |
A misrouted short-header datagram drew a stateless reset from the wrong worker. Every worker of a group shares the reset-token key, so a worker that answers a short header it holds no connection for with a reset (RFC 9000 §10.3) makes a valid reset for a sibling's live connection. Under steering that datagram does not reach it, but it can arrive where steering does not hold: in the one-worker fallback (QM-2) a stale peer of an earlier group run, or a process of the same user that joins the group and replaces its program (QM-5). Fixed: a worker drops a short header whose byte 0 names another worker of its group (below workerCount, not workerIndex); byte 0 at or above the count is no worker's, and is answered as before. |
Closed. Pinned by net_quic_worker (worker 0 of 2 drops a short header for worker 1's ID and resets one for its own, or for byte 2 or 255; unset, a listener resets what worker 0 dropped) and net_quic_prefork row 12 (a restarted worker 1 resets its own dead connection's client, and worker 0's client is untouched, row 13). |
| QM-2 | Low | std/net/prefork.ts (preforkBindUdp) |
A refused steering program would have left N workers unsteered. Without the program, SO_REUSEPORT spreads datagrams by the 4-tuple hash, so a connection's packets reach a worker that does not hold it, and under QM-1's rule are dropped; before it, they drew resets. Fixed: when the first bind with flag 4 fails, the supervisor binds one plain socket, prints nish/net/prefork: QUIC steering refused (<errno>), running one worker on stderr, and runs one worker; when even that fails it prints the bind error and binds nothing. A serviceWorker past worker 0 has no slot in that one-worker run, so the supervisor then prints so, closes the socket and starts no worker (not pinned: the steering is refused only off Linux, where no test asks for both). A later socket of the group that fails to bind closes every socket before it (not pinned: no test makes the second bind fail). |
Closed. Pinned by net_quic_prefork row 1 (one worker and the stderr line where the runtime answers -95) and tests/runtime-test.c (test_udp_steer: -95 off Linux, and every malformed flags word -22). |
| QM-3 | Low | std/net/quic.ts, steering |
A peer picks the worker its handshake lands on. The first Initial's DCID is the client's choice, so a client sets byte 0 and chooses a worker (modulo N), and a flood of Initials can be aimed at one worker. A connection moves to its own worker's IDs after the handshake, so only the handshake is aimed. | Accepted. The same holds for any steering a client's first datagram decides, including the 4-tuple hash, which a client picks by its source port. What a handshake costs is bounded per worker as in http3.md (H3-1, H3-3), and Retry (net_quic_prefork rows 15 and 16) stays on the worker that issued it. |
| QM-4 | Low | std/net/quic.ts (quicWorkerByte) |
Byte 0 of every ID names its worker. An observer learns which worker issued a connection ID, so IDs a connection moves to after migration share that byte with each other and with every other connection on that worker. RFC 9000 §9.5 asks that IDs not be linkable; one byte of N values links a connection to a group of about 1/N of the server's connections, no closer. | Accepted. It is what makes the kernel's choice possible without state, as QUIC-LB's plaintext algorithm does; the other 7 bytes are drawn fresh for each ID as before (net_quic_worker, "the other bytes are the same either way": marking changes byte 0 only). |
| QM-5 | Low | runtime/runtime-net.c (flag 4), std/net/prefork.ts (preforkBindUdp) |
Who may join the group. Linux admits a socket to a SO_REUSEPORT group when it sets the option and has the same effective uid as the group's first socket. Another user cannot join (RT-17), but any process of the same user can bind the port with SO_REUSEPORT, landing at index N, past every byte the program picks, and then attach its own program with SO_ATTACH_REUSEPORT_CBPF, which replaces the group's for every socket. It then receives whichever datagrams its program chooses, every connection's included, or scatters them across the workers, where QM-1's rule drops them. |
Accepted. A process of the same user can already signal every worker and, unless Yama's ptrace_scope is 1 or above, trace it and read its memory, the reset and Retry keys with it, so the group adds little reach it lacks. Run the server as a user of its own; the supervisor does not detect a joined socket (the kernel offers no query of a group's members). |
Decisions
- A runt goes to worker 0. A datagram too short to hold byte 0 at its offset loads 0 in the program, so it reaches socket 0. It carries no connection, so it is parsed and dropped there like any malformed datagram; the alternative, the address hash, would make one worker's drop another's.
- What steering leaks (h3-multicore.md §2.2). Byte 0 is in the clear, so an observer learns a lower bound on N (the largest byte seen, plus one) and which connections share a worker (QM-4). It learns nothing of the other 7 bytes, which are drawn per ID as before.
Properties verified
| Property | Pinned by |
|---|---|
| A short header's DCID byte 0 (offset 1) and a long header's (offset 6) pick socket (byte mod N) of the group in bind order; a datagram too short to hold it reaches socket 0 | tests/runtime-test.c, test_udp_steer |
Flag 4 without flag 1, with flag 2, with N 0, N without flag 4, and any bit outside 0xff07 answer -22; off Linux, flags 2 and 4 answer -95; a refused attach closes the socket and answers the errno |
tests/runtime-test.c, test_udp_steer; tests/cases/net_udp_calls |
| Every ID a worker issues starts with its index: the first SCID, each NEW_CONNECTION_ID and the Retry SCID; with the pair unset (-1 and 0) none is marked | net_quic_worker; net_quic_prefork rows 2, 9, 15 and 16 |
workerIndex and workerCount are both unset or 0 <= k < n <= 255; anything else panics with nish/net/quic: workerIndex <k> and workerCount <n>, not 0 <= k < n <= 255 when the listener or server is made |
net_quic_worker (the past, zero, below and wide cases) |
NISH_WORKERS above 255 panics in the supervisor before any worker starts; 255 workers start, and byte 254 reaches worker 254 |
net_quic_prefork rows 20 to 23 |
| A restarted worker gets the same socket of the group, so the kernel's choice for its byte does not move to a sibling | net_quic_prefork rows 11 and 18 |
| A connection survives its client's rebinding on the same worker, and the path is validated | net_quic_prefork rows 3 to 5 |
A worker trusts the descriptor and count in --nish-prefork-udp=<fd>,<N> as given, the same boundary as --nish-prefork-listener=; a malformed value (a missing field, a count of 0 or above 255, a negative descriptor, a third field, a non-number) gives no socket and no group, and preforkArgs strips the flag |
net_quic_worker (the flag checks) |
Steering off
net_quic_prefork's acceptance row was also run with the group bound
without flag 4 (each socket SO_REUSEPORT only): connections then reached
workers by the 4-tuple hash, and the rows that depend on the owning worker
(a rebinding client, Initials past the count, a ClientHello split across
ports, a restart's reset) failed in nearly every run. That is the behaviour
QM-2's fallback exists to avoid.

