nish/net/http-compress

Content coding for HTTP responses (RFC 9110 §8.4 and §12.5.3): reading a request's Accept-Encoding and compressing a body with Content-Encoding: gzip, for any of the three servers.

import {
  HTTP1_GZIP_FIELD_BYTES,
  HTTP_CODING_DEFLATE,
  HTTP_CODING_GZIP,
  HTTP_CODING_IDENTITY,
  HTTP_CODING_NONE,
  HTTP_COMPRESS_NOT_ACCEPTED,
  HTTP_COMPRESS_NO_FEWER_SEGMENTS,
  HTTP_COMPRESS_SENT,
  HTTP_COMPRESS_TOO_LARGE,
  HTTP_COMPRESS_TOO_LITTLE_SAVED,
  HTTP_COMPRESS_TOO_SMALL,
  HTTP_DEFAULT_MSS,
  HTTP_ENCODER_AUTO,
  HTTP_ENCODER_FIXED,
  HTTP_ENCODER_LEVEL,
  HTTP_OFFER_DEFLATE,
  HTTP_OFFER_GZIP,
  HttpCompressor,
  httpAcceptEncoding,
  httpCompressorDefault,
  httpIdentityRefused,
  httpSegments,
} from "nish/net/http-compress";
import { DEFLATE_FAST } from "nish/compress/deflate";
import { HTTP_OFFER_GZIP, HttpCompressor, httpAcceptEncoding } from "nish/net/http-compress";

const enc = new HttpCompressor(DEFLATE_FAST, 1024, 65536); // once per thread
// HTTP/1.1, in one call:
conn.respondEncoded(200, names, values, body, 0, n, enc, conn.acceptEncoding());
// HTTP/2 or HTTP/3, by hand:
const coding: i32 = httpAcceptEncoding(value, 0, toI32(value.length), HTTP_OFFER_GZIP);
const m: i32 = enc.compress(coding, body, 0, n);
if (m >= 0) {
  enc.addFields(names, values);
  h2.respond(id, 200, enc.names, enc.values, false);
  h2.writeData(id, enc.output, 0, m, true);
}
Opt-in

Nothing here runs unless the program asks: it decides which responses to compress, and HttpCompressor declines any body shorter than its minLength, longer than its maxLength, or that gzip would not make smaller, answering -1 so the program sends the bytes as they are.

Compression that pays for itself

A body is sent compressed only when it saves at least minSaving bytes and minSavingPercent percent of the body; otherwise the compressed bytes are dropped and the body goes as it is. With segmentMode set, respondEncoded also sends it compressed only when the whole response, head included, takes fewer TCP segments of mss bytes (the connection's mss if set, else the compressor's mss if set, else the MSS the server read from the socket, else HTTP_DEFAULT_MSS, 1448): a 1300-byte body that gzips to 700 still fits in one segment either way.

After each call decision says why (HTTP_COMPRESS_SENT, _NOT_ACCEPTED, _TOO_SMALL, _TOO_LARGE, _TOO_LITTLE_SAVED, _NO_FEWER_SEGMENTS), for a program to count. A response whose identity form already fits one segment is _NO_FEWER_SEGMENTS without running gzip at all, as no compression could take fewer.

Two encoders

Bodies up to fixedLimit (16 KB) go through GzipFixedCompressor: one fixed-Huffman block, a 16 KB window, no state between calls, two to four times faster than DEFLATE_FAST on 0.5 to 16 KB of JSON for output about 1.35 times larger. Larger bodies go through the GzipCompressor at the level given; encoder forces either one.

`Vary: Accept-Encoding`

Vary: Accept-Encoding (with vary, on by default) goes on every response whose coding depended on the request: one sent compressed, and one sent as it is because the request did not accept gzip. A body that is too small, too large or did not pay is sent the same to every client, so it carries no Vary. A request that refuses identity (identity;q=0) and accepts gzip is sent gzip whatever the policy (compressAnyway).

Negotiation

Negotiation (httpAcceptEncoding) follows §12.5.3: each listed coding with its q-value, x-gzip as gzip, * for every coding not listed, and identity acceptable unless it is given q=0, directly or through *.

Of the codings the program offers, the most preferred with q above 0 wins, the program's order (gzip, then deflate) breaking ties; it beats identity at an equal q, and beats an identity the field does not mention at any q. With no field at all the answer is HTTP_CODING_IDENTITY (the RFC allows any coding then, but a client that says nothing is safest sent nothing it may not read); an empty field asks for identity too. HTTP_CODING_NONE means nothing offered is acceptable and identity is refused: a server may answer 406 or send identity anyway. A malformed element is skipped. The parser allocates nothing.

Compressing allocates nothing

Compressing allocates nothing and caches nothing between responses: HttpCompressor holds a GzipCompressor, a GzipFixedCompressor, an output buffer of compressBound(maxLength) bytes and the field arrays addFields fills, all built once. HTTP_CODING_DEFLATE (zlib framing) is negotiated but not produced: a client that accepts only deflate is sent identity, which every client reads.

Functions

httpAcceptEncoding

The content coding to send for the Accept-Encoding field value buf[start .. end) among the codings offered (HTTP_OFFER_GZIP | HTTP_OFFER_DEFLATE): HTTP_CODING_GZIP, HTTP_CODING_DEFLATE, HTTP_CODING_IDENTITY or HTTP_CODING_NONE, by the rules in the module comment. A window outside buf panics.

const httpAcceptEncoding: (
  buf: u8[],
  start: i32,
  end: i32,
  offered: i32,
) => i32
bufu8[]required

the buffer holding the field value, such as the request head

starti32required

the index in buf of the value's first octet

endi32required

the index just past the value's last octet

offeredi32required

which codings the program can produce, HTTP_OFFER_* bits or 0 for none

returnsi32

the HTTP_CODING_* to send the body in, or HTTP_CODING_NONE when nothing offered is acceptable and identity is refused too

httpCompressorDefault

A compressor for the usual server: DEFLATE_FAST, bodies of 1 KB to 1 MB.

const httpCompressorDefault: () => HttpCompressor
returnsHttpCompressor

a new compressor the caller owns, one per thread, with every other setting at its default

httpIdentityRefused

Whether the Accept-Encoding value in buf[start .. end) refuses the body as it is: identity;q=0, or *;q=0 with no identity listed. A server should then send a coding the client takes even where it would not bother (HttpCompressor.compressAnyway).

const httpIdentityRefused: (
  buf: u8[],
  start: i32,
  end: i32,
) => boolean
bufu8[]required

the buffer holding the field value, such as the request head

starti32required

the index in buf of the value's first octet

endi32required

the index just past the value's last octet

returnsboolean

true when the client said it cannot take the body uncompressed

httpSegments

The number of TCP segments bytes bytes take at mss bytes a segment. A segment size of zero or less panics, since no count would mean anything.

const httpSegments: (bytes: i32, mss: i32) => i32
bytesi32required

the length of the whole response, head and body, in bytes

mssi32required

the payload one segment carries, in bytes

returnsi32

how many segments the bytes fill, the last one possibly partial; 0 for no bytes

Classes

HttpCompressor

Compresses response bodies with gzip into a buffer of its own, and builds the response fields that say so. One per thread; allocated once.

class HttpCompressor
Member
new HttpCompressor(level: i32, minLength: i32, maxLength: i32) A compressor at level (DEFLATE_FAST suits a server) for bodies of minLength to maxLength bytes. A level outside 0 to 9, a negative minLength or a maxLength below it panics.
decision: i32 The last decision, an HTTP_COMPRESS_*; HTTP_COMPRESS_NOT_ACCEPTED before any.
encoder: i32 HTTP_ENCODER_AUTO (the default), HTTP_ENCODER_FIXED or HTTP_ENCODER_LEVEL.
encoderRuns: i32 How many times compress and compressAnyway ran an encoder, from 0: a count for a test or a profile to read, never consulted by a decision. It wraps past 2^31 - 1 to -2^31 rather than panic, as a server can run that many; read it as a difference between two reads.
fixedLimit: i32 The longest body HTTP_ENCODER_AUTO gives the fixed-Huffman encoder: 16384 by default.
minSaving: i32 The least a compressed body must save to be sent: minSaving bytes and minSavingPercent percent of the body, whichever is more. 1 and 0 by default (any saving at all); a value below those counts as them.
minSavingPercent: i32 The percent of the body a compressed body must save, as well as minSaving bytes: 0 by default, 100 or more sends nothing compressed.
mss: i32 The TCP segment size segmentMode counts in, in bytes, when the program sets it: 0 (the default) or less means unset, so a connection's socket MSS is used, or HTTP_DEFAULT_MSS where there is none. A value set here wins over the socket's; a connection's own mss wins over it.
readonly names: u8[][] The fields addFields built: the program's, then Content-Encoding and Vary.
readonly output: u8[] Where compress writes; valid until the next call.
segmentMode: boolean Whether respondEncoded also sends the compressed response only when, head included, it takes fewer TCP segments of mss bytes than the body as it is; off by default. Segments are the connection's own mss if the program set it, else this compressor's mss if set, else the socket's MSS the server read on accept, else HTTP_DEFAULT_MSS.
readonly values: u8[][] The value of each field in names, at the same index; pass both to respond.
vary: boolean Whether addFields adds Vary: Accept-Encoding (RFC 9110 §12.5.5), which a cache needs whenever the same resource may be sent either way. On by default; turn it off when the program sets its own Vary.
addFields(names: u8[][], values: u8[][]): void Sets names / values to the program's fields followed by Content-Encoding: gzip and, when vary, Vary: Accept-Encoding. The arrays are reused, so after the first few calls this allocates nothing; they hold until the next call.
compress(coding: i32, body: u8[], off: i32, len: i32): i32 Compresses body[off .. off + len) into output when the length is in [minLength, maxLength], coding is HTTP_CODING_GZIP and gzip saves what minSaving and minSavingPercent ask; answers the compressed length, or -1 to send the body as it is, and sets decision either way. encoder picks the encoder; the fixed-Huffman one stops as soon as its output passes what would be kept. A window outside body panics.
compressAnyway(coding: i32, body: u8[], off: i32, len: i32): i32 compress for a client that refuses identity (httpIdentityRefused): gzip whatever the body's length, the saving or the segments, so long as it is no longer than maxLength. Answers the compressed length, or -1 when coding is not gzip or the body is too long (sent as it is, which RFC 9110 §12.5.3 allows).
fewerSegments(identityBytes: i32, encodedBytes: i32, mss: i32): boolean segmentMode's rule, for a response of identityBytes bytes sent as it is or encodedBytes compressed, heads included, at mss bytes a segment (this.mss when 0 or less, and HTTP_DEFAULT_MSS when both are, as segmentSize reads it): whether the compressed one takes fewer segments. When it does not, decision becomes HTTP_COMPRESS_NO_FEWER_SEGMENTS. respondEncoded calls it; a program driving HTTP/2 or HTTP/3 by hand may with its own head sizes.
http1IdentityHead(encodedHead: i32, encodedLength: i32, length: i32): i32 The bytes of the HTTP/1.1 head a length-byte body sent as it is would get, from encodedHead, that of the same response sent compressed as encodedLength bytes with the fields addFields built: less Content-Encoding: gzip, less Vary (a body the segment count kept as it is would go out so to any client), and the other Content-Length.
identityFields(names: u8[][], values: u8[][]): void Sets names / values to the program's fields and, when vary and the last decision varies, Vary: Accept-Encoding: the fields of a body sent as it is from a resource that is sometimes compressed.
leastSaving(len: i32): i32 The bytes a compressed len-byte body must save: minSaving and minSavingPercent, at least 1.
oneSegment(coding: i32, len: i32, mss: i32): boolean segmentMode's shortcut, before any encoder runs: whether a len-byte body that compress would try (its length in [minLength, maxLength] and coding gzip) is no longer than one segment of mss bytes (as fewerSegments reads it). Sent as it is, such a response takes one segment once its head fits beside it, and no compression takes fewer, so decision becomes HTTP_COMPRESS_NO_FEWER_SEGMENTS and the caller checks the head: when it does not fit, the caller goes on to compress, which decides again. Anything else answers false and leaves decision alone.
varies(): boolean Whether the last decision depended on the request's Accept-Encoding, so the response needs Vary: Accept-Encoding: a body sent compressed, or one sent as it is because the client did not take gzip. A body the lengths or the saving kept as it is would go out the same to any client.

Limits

1,448

HTTP_DEFAULT_MSS: i32

The maximum segment size segmentMode counts with unless told the connection's: a 1500-byte Ethernet MTU less 20 bytes of IPv4 header, 20 of TCP header and 12 of the timestamp option.

Constants

24

HTTP1_GZIP_FIELD_BYTES: i32

The bytes content-encoding: gzip adds to an HTTP/1.1 head: name, value, ": " and CRLF.

2

HTTP_CODING_DEFLATE: i32

deflate: the zlib format (RFC 9110 §8.4.1.2).

1

HTTP_CODING_GZIP: i32

gzip (RFC 9110 §8.4.1.3), or its alias x-gzip.

0

HTTP_CODING_IDENTITY: i32

No content coding: the body as it is.

-1

HTTP_CODING_NONE: i32

Nothing offered is acceptable, identity included (§12.5.3: 406, or identity anyway).

1

HTTP_COMPRESS_NOT_ACCEPTED: i32

The client did not accept gzip, so the body went as it is; the default before any decision.

5

HTTP_COMPRESS_NO_FEWER_SEGMENTS: i32

Under segmentMode, compressing would not have saved a TCP segment.

0

HTTP_COMPRESS_SENT: i32

What HttpCompressor.compress (and respondEncoded's segment check) last decided, in HttpCompressor.decision, for a program to count: the body went out compressed; the client did not take gzip; it was shorter than minLength; longer than maxLength; gzip did not save minSaving bytes and minSavingPercent percent; or the compressed response would not have taken fewer TCP segments (segmentMode), which includes an identity response that already fits one segment, decided without compressing.

3

HTTP_COMPRESS_TOO_LARGE: i32

The body was longer than maxLength, more than output was sized for.

4

HTTP_COMPRESS_TOO_LITTLE_SAVED: i32

Gzip ran but did not save minSaving bytes and minSavingPercent percent, or the saving asked for was the whole body, so no encoder ran.

2

HTTP_COMPRESS_TOO_SMALL: i32

The body was shorter than minLength, too small to be worth compressing.

0

HTTP_ENCODER_AUTO: i32

HttpCompressor.encoder: by the body's length (the fixed-Huffman encoder up to fixedLimit bytes, the constructor's level above), the fixed-Huffman encoder always, or the constructor's level always.

1

HTTP_ENCODER_FIXED: i32

The fixed-Huffman encoder for every body compress takes: fastest, at some cost in size on large bodies. compressAnyway falls back to the GzipCompressor when it fails.

2

HTTP_ENCODER_LEVEL: i32

Always the GzipCompressor at the constructor's level, whatever the body's length.

2

HTTP_OFFER_DEFLATE: i32

The offered bit for deflate, which is negotiated but never produced here.

1

HTTP_OFFER_GZIP: i32

httpAcceptEncoding's offered bits: the program can send gzip, deflate.