nish/net/http-compress
Content coding for HTTP responses (RFC 9110 §8.4 and §12.5.3): reading a request's Accept-Encoding and compressing a body with Content-Encoding: gzip, for any of the three servers.
import {
HTTP1_GZIP_FIELD_BYTES,
HTTP_CODING_DEFLATE,
HTTP_CODING_GZIP,
HTTP_CODING_IDENTITY,
HTTP_CODING_NONE,
HTTP_COMPRESS_NOT_ACCEPTED,
HTTP_COMPRESS_NO_FEWER_SEGMENTS,
HTTP_COMPRESS_SENT,
HTTP_COMPRESS_TOO_LARGE,
HTTP_COMPRESS_TOO_LITTLE_SAVED,
HTTP_COMPRESS_TOO_SMALL,
HTTP_DEFAULT_MSS,
HTTP_ENCODER_AUTO,
HTTP_ENCODER_FIXED,
HTTP_ENCODER_LEVEL,
HTTP_OFFER_DEFLATE,
HTTP_OFFER_GZIP,
HttpCompressor,
httpAcceptEncoding,
httpCompressorDefault,
httpIdentityRefused,
httpSegments,
} from "nish/net/http-compress";
import { DEFLATE_FAST } from "nish/compress/deflate";
import { HTTP_OFFER_GZIP, HttpCompressor, httpAcceptEncoding } from "nish/net/http-compress";
const enc = new HttpCompressor(DEFLATE_FAST, 1024, 65536); // once per thread
// HTTP/1.1, in one call:
conn.respondEncoded(200, names, values, body, 0, n, enc, conn.acceptEncoding());
// HTTP/2 or HTTP/3, by hand:
const coding: i32 = httpAcceptEncoding(value, 0, toI32(value.length), HTTP_OFFER_GZIP);
const m: i32 = enc.compress(coding, body, 0, n);
if (m >= 0) {
enc.addFields(names, values);
h2.respond(id, 200, enc.names, enc.values, false);
h2.writeData(id, enc.output, 0, m, true);
}
httpAcceptEncoding
The content coding to send for the Accept-Encoding field value buf[start .. end) among the codings offered (HTTP_OFFER_GZIP | HTTP_OFFER_DEFLATE): HTTP_CODING_GZIP, HTTP_CODING_DEFLATE, HTTP_CODING_IDENTITY or HTTP_CODING_NONE, by the rules in the module comment.
httpCompressorDefault
A compressor for the usual server: DEFLATE_FAST, bodies of 1 KB to 1 MB.
httpIdentityRefused
Whether the Accept-Encoding value in buf[start .. end) refuses the body as it is: identity;q=0, or *;q=0 with no identity listed.
httpSegments
The number of TCP segments bytes bytes take at mss bytes a segment.
HttpCompressor
Compresses response bodies with gzip into a buffer of its own, and builds the response fields that say so.
Opt-in
Nothing here runs unless the program asks: it decides which
responses to compress, and HttpCompressor declines any body shorter than
its minLength, longer than its maxLength, or that gzip would not make
smaller, answering -1 so the program sends the bytes as they are.
Compression that pays for itself
A body is sent compressed only when it
saves at least minSaving bytes and minSavingPercent percent of the body;
otherwise the compressed bytes are dropped and the body goes as it is. With
segmentMode set, respondEncoded also sends it compressed only when the
whole response, head included, takes fewer TCP segments of mss bytes (the
connection's mss if set, else the compressor's mss if set, else the MSS
the server read from the socket, else HTTP_DEFAULT_MSS, 1448): a
1300-byte body that gzips to 700 still fits in one segment either way.
After each call decision says why (HTTP_COMPRESS_SENT, _NOT_ACCEPTED,
_TOO_SMALL, _TOO_LARGE, _TOO_LITTLE_SAVED, _NO_FEWER_SEGMENTS), for a
program to count. A response whose identity form already fits one segment is
_NO_FEWER_SEGMENTS without running gzip at all, as no compression could
take fewer.
Two encoders
Bodies up to fixedLimit (16 KB) go through
GzipFixedCompressor: one fixed-Huffman block, a 16 KB window, no state
between calls, two to four times faster than DEFLATE_FAST on 0.5 to
16 KB of JSON for output about 1.35 times larger. Larger bodies go through
the GzipCompressor at the level given; encoder forces either one.
`Vary: Accept-Encoding`
Vary: Accept-Encoding (with vary, on by default) goes on every
response whose coding depended on the request: one sent compressed, and
one sent as it is because the request did not accept gzip. A body that is
too small, too large or did not pay is sent the same to every client, so
it carries no Vary. A request that refuses identity (identity;q=0)
and accepts gzip is sent gzip whatever the policy (compressAnyway).
Negotiation
Negotiation (httpAcceptEncoding) follows §12.5.3: each listed coding
with its q-value, x-gzip as gzip, * for every coding not listed, and
identity acceptable unless it is given q=0, directly or through *.
Of the codings the program offers, the most preferred with q above 0 wins,
the program's order (gzip, then deflate) breaking ties; it beats identity
at an equal q, and beats an identity the field does not mention at any q.
With no field at all the answer is HTTP_CODING_IDENTITY (the RFC allows any
coding then, but a client that says nothing is safest sent nothing it may not
read); an empty field asks for identity too. HTTP_CODING_NONE means nothing
offered is acceptable and identity is refused: a server may answer 406 or
send identity anyway. A malformed element is skipped. The parser allocates
nothing.
Compressing allocates nothing
Compressing allocates nothing and caches nothing between responses:
HttpCompressor holds a GzipCompressor, a GzipFixedCompressor, an
output buffer of compressBound(maxLength) bytes and
the field arrays addFields fills, all built once.
HTTP_CODING_DEFLATE (zlib framing) is negotiated but not produced: a
client that accepts only deflate is sent identity, which every client
reads.
Functions
httpAcceptEncoding
The content coding to send for the Accept-Encoding field value
buf[start .. end) among the codings offered (HTTP_OFFER_GZIP | HTTP_OFFER_DEFLATE): HTTP_CODING_GZIP, HTTP_CODING_DEFLATE,
HTTP_CODING_IDENTITY or HTTP_CODING_NONE, by the rules in the module
comment. A window outside buf panics.
const httpAcceptEncoding: (
buf: u8[],
start: i32,
end: i32,
offered: i32,
) => i32
bufu8[]requiredthe buffer holding the field value, such as the request head
starti32requiredthe index in buf of the value's first octet
endi32requiredthe index just past the value's last octet
offeredi32required
which codings the program can produce, HTTP_OFFER_* bits
or 0 for none
returnsi32
the HTTP_CODING_* to send the body in, or HTTP_CODING_NONE
when nothing offered is acceptable and identity is refused too
httpCompressorDefault
A compressor for the usual server: DEFLATE_FAST, bodies of 1 KB to 1 MB.
const httpCompressorDefault: () => HttpCompressor
returnsHttpCompressora new compressor the caller owns, one per thread, with every other setting at its default
httpIdentityRefused
Whether the Accept-Encoding value in buf[start .. end) refuses the
body as it is: identity;q=0, or *;q=0 with no identity listed. A
server should then send a coding the client takes even where it would not
bother (HttpCompressor.compressAnyway).
const httpIdentityRefused: (
buf: u8[],
start: i32,
end: i32,
) => boolean
bufu8[]requiredthe buffer holding the field value, such as the request head
starti32requiredthe index in buf of the value's first octet
endi32requiredthe index just past the value's last octet
returnsbooleantrue when the client said it cannot take the body uncompressed
httpSegments
The number of TCP segments bytes bytes take at mss bytes a segment.
A segment size of zero or less panics, since no count would mean anything.
const httpSegments: (bytes: i32, mss: i32) => i32
bytesi32requiredthe length of the whole response, head and body, in bytes
mssi32requiredthe payload one segment carries, in bytes
returnsi32how many segments the bytes fill, the last one possibly partial; 0 for no bytes
Classes
HttpCompressor
Compresses response bodies with gzip into a buffer of its own, and builds the response fields that say so. One per thread; allocated once.
class HttpCompressor
| Member | |
|---|---|
new HttpCompressor(level: i32, minLength: i32, maxLength: i32) |
A compressor at level (DEFLATE_FAST suits a server) for bodies of minLength to maxLength bytes. A level outside 0 to 9, a negative minLength or a maxLength below it panics. |
decision: i32 |
The last decision, an HTTP_COMPRESS_*; HTTP_COMPRESS_NOT_ACCEPTED before any. |
encoder: i32 |
HTTP_ENCODER_AUTO (the default), HTTP_ENCODER_FIXED or HTTP_ENCODER_LEVEL. |
encoderRuns: i32 |
How many times compress and compressAnyway ran an encoder, from 0: a count for a test or a profile to read, never consulted by a decision. It wraps past 2^31 - 1 to -2^31 rather than panic, as a server can run that many; read it as a difference between two reads. |
fixedLimit: i32 |
The longest body HTTP_ENCODER_AUTO gives the fixed-Huffman encoder: 16384 by default. |
minSaving: i32 |
The least a compressed body must save to be sent: minSaving bytes and minSavingPercent percent of the body, whichever is more. 1 and 0 by default (any saving at all); a value below those counts as them. |
minSavingPercent: i32 |
The percent of the body a compressed body must save, as well as minSaving bytes: 0 by default, 100 or more sends nothing compressed. |
mss: i32 |
The TCP segment size segmentMode counts in, in bytes, when the program sets it: 0 (the default) or less means unset, so a connection's socket MSS is used, or HTTP_DEFAULT_MSS where there is none. A value set here wins over the socket's; a connection's own mss wins over it. |
readonly names: u8[][] |
The fields addFields built: the program's, then Content-Encoding and Vary. |
readonly output: u8[] |
Where compress writes; valid until the next call. |
segmentMode: boolean |
Whether respondEncoded also sends the compressed response only when, head included, it takes fewer TCP segments of mss bytes than the body as it is; off by default. Segments are the connection's own mss if the program set it, else this compressor's mss if set, else the socket's MSS the server read on accept, else HTTP_DEFAULT_MSS. |
readonly values: u8[][] |
The value of each field in names, at the same index; pass both to respond. |
vary: boolean |
Whether addFields adds Vary: Accept-Encoding (RFC 9110 §12.5.5), which a cache needs whenever the same resource may be sent either way. On by default; turn it off when the program sets its own Vary. |
addFields(names: u8[][], values: u8[][]): void |
Sets names / values to the program's fields followed by Content-Encoding: gzip and, when vary, Vary: Accept-Encoding. The arrays are reused, so after the first few calls this allocates nothing; they hold until the next call. |
compress(coding: i32, body: u8[], off: i32, len: i32): i32 |
Compresses body[off .. off + len) into output when the length is in [minLength, maxLength], coding is HTTP_CODING_GZIP and gzip saves what minSaving and minSavingPercent ask; answers the compressed length, or -1 to send the body as it is, and sets decision either way. encoder picks the encoder; the fixed-Huffman one stops as soon as its output passes what would be kept. A window outside body panics. |
compressAnyway(coding: i32, body: u8[], off: i32, len: i32): i32 |
compress for a client that refuses identity (httpIdentityRefused): gzip whatever the body's length, the saving or the segments, so long as it is no longer than maxLength. Answers the compressed length, or -1 when coding is not gzip or the body is too long (sent as it is, which RFC 9110 §12.5.3 allows). |
fewerSegments(identityBytes: i32, encodedBytes: i32, mss: i32): boolean |
segmentMode's rule, for a response of identityBytes bytes sent as it is or encodedBytes compressed, heads included, at mss bytes a segment (this.mss when 0 or less, and HTTP_DEFAULT_MSS when both are, as segmentSize reads it): whether the compressed one takes fewer segments. When it does not, decision becomes HTTP_COMPRESS_NO_FEWER_SEGMENTS. respondEncoded calls it; a program driving HTTP/2 or HTTP/3 by hand may with its own head sizes. |
http1IdentityHead(encodedHead: i32, encodedLength: i32, length: i32): i32 |
The bytes of the HTTP/1.1 head a length-byte body sent as it is would get, from encodedHead, that of the same response sent compressed as encodedLength bytes with the fields addFields built: less Content-Encoding: gzip, less Vary (a body the segment count kept as it is would go out so to any client), and the other Content-Length. |
identityFields(names: u8[][], values: u8[][]): void |
Sets names / values to the program's fields and, when vary and the last decision varies, Vary: Accept-Encoding: the fields of a body sent as it is from a resource that is sometimes compressed. |
leastSaving(len: i32): i32 |
The bytes a compressed len-byte body must save: minSaving and minSavingPercent, at least 1. |
oneSegment(coding: i32, len: i32, mss: i32): boolean |
segmentMode's shortcut, before any encoder runs: whether a len-byte body that compress would try (its length in [minLength, maxLength] and coding gzip) is no longer than one segment of mss bytes (as fewerSegments reads it). Sent as it is, such a response takes one segment once its head fits beside it, and no compression takes fewer, so decision becomes HTTP_COMPRESS_NO_FEWER_SEGMENTS and the caller checks the head: when it does not fit, the caller goes on to compress, which decides again. Anything else answers false and leaves decision alone. |
varies(): boolean |
Whether the last decision depended on the request's Accept-Encoding, so the response needs Vary: Accept-Encoding: a body sent compressed, or one sent as it is because the client did not take gzip. A body the lengths or the saving kept as it is would go out the same to any client. |
Limits
1,448
HTTP_DEFAULT_MSS: i32
The maximum segment size segmentMode counts with unless told the
connection's: a 1500-byte Ethernet MTU less 20 bytes of IPv4 header, 20 of
TCP header and 12 of the timestamp option.
Constants
24
HTTP1_GZIP_FIELD_BYTES: i32
The bytes content-encoding: gzip adds to an HTTP/1.1 head: name, value, ": " and CRLF.
2
HTTP_CODING_DEFLATE: i32
deflate: the zlib format (RFC 9110 §8.4.1.2).
1
HTTP_CODING_GZIP: i32
gzip (RFC 9110 §8.4.1.3), or its alias x-gzip.
0
HTTP_CODING_IDENTITY: i32
No content coding: the body as it is.
-1
HTTP_CODING_NONE: i32
Nothing offered is acceptable, identity included (§12.5.3: 406, or identity anyway).
1
HTTP_COMPRESS_NOT_ACCEPTED: i32
The client did not accept gzip, so the body went as it is; the default before any decision.
5
HTTP_COMPRESS_NO_FEWER_SEGMENTS: i32
Under segmentMode, compressing would not have saved a TCP segment.
0
HTTP_COMPRESS_SENT: i32
What HttpCompressor.compress (and respondEncoded's segment check) last
decided, in HttpCompressor.decision, for a program to count: the body
went out compressed; the client did not take gzip; it was shorter than
minLength; longer than maxLength; gzip did not save minSaving bytes
and minSavingPercent percent; or the compressed response would not have
taken fewer TCP segments (segmentMode), which includes an identity
response that already fits one segment, decided without compressing.
3
HTTP_COMPRESS_TOO_LARGE: i32
The body was longer than maxLength, more than output was sized for.
4
HTTP_COMPRESS_TOO_LITTLE_SAVED: i32
Gzip ran but did not save minSaving bytes and minSavingPercent percent,
or the saving asked for was the whole body, so no encoder ran.
2
HTTP_COMPRESS_TOO_SMALL: i32
The body was shorter than minLength, too small to be worth compressing.
0
HTTP_ENCODER_AUTO: i32
HttpCompressor.encoder: by the body's length (the fixed-Huffman encoder up
to fixedLimit bytes, the constructor's level above), the fixed-Huffman
encoder always, or the constructor's level always.
1
HTTP_ENCODER_FIXED: i32
The fixed-Huffman encoder for every body compress takes: fastest, at some
cost in size on large bodies. compressAnyway falls back to the
GzipCompressor when it fails.
2
HTTP_ENCODER_LEVEL: i32
Always the GzipCompressor at the constructor's level, whatever the body's length.
2
HTTP_OFFER_DEFLATE: i32
The offered bit for deflate, which is negotiated but never produced here.
1
HTTP_OFFER_GZIP: i32
httpAcceptEncoding's offered bits: the program can send gzip, deflate.

