nish/compress/gzip

The gzip file format (RFC 1952) around nish/compress/deflate: a ten-byte header, the DEFLATE stream, and a trailer of the CRC-32 and the length of the data mod 2^32. What an HTTP Content-Encoding: gzip body is.

import {
  GZIP_END,
  GZIP_ERR_CRC,
  GZIP_ERR_HEADER,
  GZIP_ERR_SIZE,
  GZIP_NEED_INPUT,
  GzipCompressor,
  GzipDecompressor,
  GzipFixedCompressor,
  GzipReader,
  GzipWriter,
  gzipCompress,
  gzipDecompress,
  gzipFixedBound,
} from "nish/compress/gzip";
import { DEFLATE_FAST, compressBound } from "nish/compress/deflate";
import { GzipCompressor, GzipDecompressor } from "nish/compress/gzip";

const gz = new GzipCompressor(DEFLATE_FAST); // once per thread
const out: u8[] = new Array<u8>(compressBound(8192)); // once
const end: i32 = gz.compress(body, 0, n, out, 0); // -1: no room

const gunzip = new GzipDecompressor();
const got: i32 = gunzip.decompress(src, 0, len, back, 0, 1 << 20); // < 0: an error
One shot

GzipCompressor and GzipDecompressor own every table a call needs and allocate nothing per call; gzipCompress and gzipDecompress build one for a single call. The header written carries no name, no comment and a zero modification time, XFL 4 at level 1 and 2 at 9, and OS 255 ("unknown"), so the same input gives the same bytes on every machine.

Reading is strict

A header with a method other than 8 or a reserved flag bit is GZIP_ERR_HEADER; FEXTRA, FNAME, FCOMMENT and FHCRC are read and skipped, the header CRC checked. A wrong CRC-32 is GZIP_ERR_CRC, a wrong length GZIP_ERR_SIZE. Members may follow one another (§2.2) and are decompressed one after another into the same output. Bytes after the last member are read as another member's header, so trailing garbage is refused, not ignored: two or more bytes without the 1f 8b magic are GZIP_ERR_HEADER, while a single byte, or fewer than 10 that start with 1f 8b, is DEFLATE_ERR_TRUNCATED. The limit bounds the whole output.

Streaming

GzipWriter compresses a body that arrives in pieces and GzipReader decompresses one: see each class.

Written from RFC 1952.

Functions

gzipCompress

GzipCompressor.compress at level, building the compressor for this call alone (about 330 KB, allocated and filled): keep a GzipCompressor for anything repeated.

const gzipCompress: (
  src: u8[],
  off: i32,
  len: i32,
  out: u8[],
  at: i32,
  level: i32,
) => i32
srcu8[]required

The array that holds the data to compress.

offi32required

Where in src the data starts.

leni32required

How many bytes of data to compress.

outu8[]required

The caller's array the member is written into.

ati32required

Where in out the member starts.

leveli32required

How hard to compress, 0 to 9; another level panics.

returnsi32

The index in out one past the member's last byte, or DEFLATE_NO_ROOM, as GzipCompressor.compress answers.

gzipDecompress

GzipDecompressor.decompress, building the decompressor for this call alone.

const gzipDecompress: (
  src: u8[],
  off: i32,
  len: i32,
  out: u8[],
  at: i32,
  limit: i32,
) => i32
srcu8[]required

The array that holds the gzip data, one member or several.

offi32required

Where in src the first member starts.

leni32required

How many bytes the members are.

outu8[]required

The caller's array the data is written into.

ati32required

Where in out the data starts.

limiti32required

The most bytes of data to accept over all the members.

returnsi32

The index in out one past the data's last byte, or a negative code, as GzipDecompressor.decompress answers.

gzipFixedBound

The most bytes GzipFixedCompressor.compress writes for n bytes of input.

const gzipFixedBound: (n: i32) => i32
ni32required

The length of the input, in bytes; a negative n answers the bound of 0.

returnsi32

A size in bytes, not a position: a room of this many always holds the member.

Classes

GzipCompressor

A gzip compressor at one level: a Deflater and the CRC-32 tables, allocated once and reused by every call. Not shared between threads.

class GzipCompressor
Member
new GzipCompressor(level: i32) A compressor at level, 0 to 9 (DEFLATE_FAST, DEFLATE_DEFAULT, ...); another level panics.
compress(src: u8[], off: i32, len: i32, out: u8[], at: i32): i32 Compresses src[off .. off + len) into out from at as one gzip member, and answers where it ends, or DEFLATE_NO_ROOM (-1) when out cannot hold it; compressBound(len) bytes always can. A window outside its array panics. Allocates nothing.

GzipDecompressor

A gzip decompressor: an Inflater and the CRC-32 tables, allocated once. Not shared between threads.

class GzipDecompressor
Member
new GzipDecompressor() Builds the tables.
decompress(src: u8[], off: i32, len: i32, out: u8[], at: i32, limit: i32): i32 Decompresses every member of src[off .. off + len) into out from at, writing at most limit bytes and never past out's end, and answers where the output ends; or DEFLATE_ERR_OUTPUT_LIMIT when the data is longer, GZIP_ERR_HEADER, GZIP_ERR_CRC, GZIP_ERR_SIZE, DEFLATE_ERR_TRUNCATED or another DEFLATE_ERR_* for a malformed stream. An empty input is truncated. A window outside its array, or a negative limit, panics. Allocates nothing.

GzipFixedCompressor

A gzip compressor for short inputs: a FixedDeflater (one fixed-Huffman block, 16 KB window) and the CRC-32 tables, allocated once. Each call depends on its input alone. Not shared between threads.

class GzipFixedCompressor
Member
new GzipFixedCompressor() Builds the tables.
compress(src: u8[], off: i32, len: i32, out: u8[], at: i32, room: i32): i32 Compresses src[off .. off + len) into out[at .. at + room) as one gzip member, and answers where it ends, or DEFLATE_NO_ROOM (-1) as soon as it would need more than room bytes, so a caller passing the most it will take stops early on an input that does not shrink. gzipFixedBound(len) always fits. A window outside its array panics. Allocates nothing.

GzipReader

A streaming gzip decompressor, for a body that arrives in pieces or does not fit in memory. Sans-IO in the nish/net/postgres shape: copy input into inputBuffer() at inputEnd() and commit it (or feed a window), then call read until it answers GZIP_NEED_INPUT; at the end of the input, endInput(). read(out, at, max) copies up to max decoded bytes and answers how many, GZIP_NEED_INPUT, GZIP_END once the stream and its trailer are through, or a negative error, final. A total of more than limit decoded bytes is DEFLATE_ERR_OUTPUT_LIMIT. Concatenated members are read as one body, as by GzipDecompressor. About 100 KB, allocated once; start begins the next body.

class GzipReader
Member
new GzipReader(limit: i64) A reader that refuses more than limit decoded bytes in all; a negative limit panics. It is ready for a first body without a call to start.
commit(n: i32): void Takes n bytes copied into inputBuffer() at inputEnd(); more than inputRoom() panics.
endInput(): void Says that no input follows what has been given.
feed(src: u8[], off: i32, len: i32): i32 Copies as much of src[off .. off + len) as fits and answers how much. A window outside src panics.
inputBuffer(): u8[] The buffer to copy input into, from inputEnd() for at most inputRoom() bytes; then commit how many were copied. It may first move the input not yet decoded to the front, so read inputEnd() after this call.
inputEnd(): i32 Where new input goes in inputBuffer().
inputRoom(): i32 How many bytes inputBuffer() has room for, from inputEnd(). Like inputBuffer(), it may first move the input not yet decoded to the front.
read(out: u8[], at: i32, max: i32): i32 Copies up to max decoded bytes into out from at and answers how many (at least 1; max is at least 1), or GZIP_NEED_INPUT when it needs more input first, GZIP_END once the body has ended, or a negative error, after which it answers that error again until start. A window outside out panics.
start(): void Begins a new body, dropping any input and output left over.

GzipWriter

A streaming gzip compressor, for a body that does not fit in one buffer: write pieces, flush to make everything so far decodable, finish once. It buffers up to 64 KB of input (and 32 KB of history to match against), compressing whenever that fills, so output arrives in bursts. Every call writes into a caller window and answers where its output ends, or DEFLATE_NO_ROOM when the window is shorter than bound(len), the most that call can produce; nothing is consumed then. About 430 KB, allocated once; start begins the next body.

class GzipWriter
Member
new GzipWriter(level: i32) A writer at level, 0 to 9; another level panics. It is ready for a first body without a call to start.
bound(len: i32): i32 The most bytes a write of len bytes, or a flush or finish after it, can produce. Less than 64 KB is pending before the call, and the call compresses once per 64 KB pending: deflateBound of all of it, with the constant part of deflateBound again for each compression after the first.
finish(out: u8[], at: i32): i32 Compresses everything pending as the final block and writes the trailer; answers the new end of out, or DEFLATE_NO_ROOM below bound(0). start begins another body; until then write, flush and finish panic, as does a window outside out.
flush(out: u8[], at: i32): i32 Compresses everything pending and ends with an empty stored block, so a reader can decode every byte written so far (zlib's Z_SYNC_FLUSH). Answers the new end of out, or DEFLATE_NO_ROOM below bound(0). Flushing after finish panics, as does a window outside out.
start(): void Begins a new body: the next output starts with a header.
write(src: u8[], off: i32, len: i32, out: u8[], at: i32): i32 Takes src[off .. off + len) and answers where the output it produced in out from at ends (often at: input waits until 64 KB are pending), or DEFLATE_NO_ROOM when out has less than bound(len) bytes from at. Writing after finish panics, as does a window outside its array.

Constants

-100

GZIP_END: i32

GzipReader: the stream has ended and every byte of it was read.

-10

GZIP_ERR_CRC: i32

The trailer's CRC-32 is not the CRC-32 of the data.

-9

GZIP_ERR_HEADER: i32

A gzip header that is not one: the wrong magic, a method other than DEFLATE, a reserved flag, or a bad header CRC.

-11

GZIP_ERR_SIZE: i32

The trailer's length is not the data's length mod 2^32.

0

GZIP_NEED_INPUT: i32

GzipReader: no more output yet; give it input.