`deflate` — deflate compression & decompression¶

This module allows compression and decompression of binary data with the DEFLATE algorithm (commonly used in the zlib library and gzip archiver). Added in MicroPython v1.21.

Availability on OpenMV-supported boards:

Board	Decompression	Compression
OpenMV Cam N6	Yes	No
OpenMV AE3	Yes	Yes
OpenMV Cam RT1062	Yes	Yes
OpenMV Cam Pure Thermal	Yes	No
OpenMV Cam M4	Yes	No
OpenMV Cam M7	Yes	No
OpenMV Cam H7	Yes	No
OpenMV Cam H7 Plus	Yes	No
Arduino Giga	Yes	No
Arduino Portenta H7	Yes	No
Arduino Nicla Vision	Yes	No
Arduino Nano 33 BLE Sense	Yes	No
Arduino Nano RP2040 Connect	Yes	No

Classes¶

class deflate.DeflateIO(stream: Any, format: int = AUTO, wbits: int = 0, close: bool = False, /)¶

This class can be used to wrap a stream which is any stream-like object such as a file, socket, or stream (including io.BytesIO). It is itself a stream and implements the standard read/readinto/write/close methods.

The stream must be a blocking stream. Non-blocking streams are currently not supported.

The format can be set to any of the constants defined below, and defaults to AUTO which for decompressing will auto-detect gzip or zlib streams, and for compressing it will generate a raw stream.

The wbits parameter sets the base-2 logarithm of the DEFLATE dictionary window size. So for example, setting wbits to 10 sets the window size to 1024 bytes. Valid values are 5 to 15 inclusive (corresponding to window sizes of 32 to 32k bytes).

If wbits is set to 0 (the default), then for compression a window size of 256 bytes will be used (as if wbits was set to 8). For decompression, it depends on the format:

RAW will use 256 bytes (corresponding to wbits set to 8).
ZLIB (or AUTO with zlib detected) will use the value from the zlib header.
GZIP (or AUTO with gzip detected) will use 32 kilobytes (corresponding to wbits set to 15).

See the window size notes below for more information about the window size, zlib, and gzip streams.

If close is set to True then the underlying stream will be closed automatically when the deflate.DeflateIO stream is closed. This is useful if you want to return a deflate.DeflateIO stream that wraps another stream and not have the caller need to know about managing the underlying stream.

If compression is enabled, a given deflate.DeflateIO instance supports both reading and writing. For example, a bidirectional stream like a socket can be wrapped, which allows for compression/decompression in both directions.

Constants¶

The four format constants select the framing applied around the raw deflate bit-stream.

deflate.AUTO: int¶: For decompression, auto-detect the input format by inspecting the stream’s first bytes (zlib or gzip). For compression, generate a raw deflate stream with no header or trailer (equivalent to RAW).

deflate.RAW: int¶: A raw deflate stream (no header, no trailer, no checksum). Because the stream contains no metadata, the decompressor cannot recover the window size from the data, so wbits should be set explicitly when decompressing – otherwise the default 256-byte window may be too small.

deflate.ZLIB: int¶: A zlib-wrapped deflate stream as defined by RFC 1950: a 2-byte header that records the window size, the deflate payload, and a trailing Adler-32 checksum. Compact and self-describing; well suited to embedded use.

deflate.GZIP: int¶: A gzip-wrapped deflate stream as defined by RFC 1952: a header with optional filename/timestamp metadata, the deflate payload, and a trailing CRC-32 plus uncompressed length. This is the format produced by the gzip command-line tool and by gzip.GzipFile. The header does not record the window size, so the decompressor must assume 32 KiB unless wbits is set.

Examples¶

A typical use case for deflate.DeflateIO is to read or write a compressed file from storage:

import deflate

# Writing a zlib-compressed stream (uses the default window size of 256 bytes).
with open("data.z", "wb") as f:
    with deflate.DeflateIO(f, deflate.ZLIB) as d:
        # Use d.write(...) etc

# Reading a zlib-compressed stream (auto-detect window size).
with open("data.z", "rb") as f:
    with deflate.DeflateIO(f, deflate.ZLIB) as d:
        # Use d.read(), d.readinto(), etc.

Because deflate.DeflateIO is a stream, it can be used for example with json.dump() and json.load() (and any other places streams can be used):

import deflate, json

# Write a dictionary as JSON in gzip format, with a
# small (64 byte) window size.
config = { ... }
with open("config.gz", "wb") as f:
    with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
        json.dump(config, f)

# Read back that dictionary.
with open("config.gz", "rb") as f:
    with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
        config = json.load(f)

If your source data is not in a stream format, you can use io.BytesIO to turn it into a stream suitable for use with deflate.DeflateIO:

import deflate, io

# Decompress a bytes/bytearray value.
compressed_data = get_data_z()
with deflate.DeflateIO(io.BytesIO(compressed_data), deflate.ZLIB) as d:
    decompressed_data = d.read()

# Compress a bytes/bytearray value.
uncompressed_data = get_data()
stream = io.BytesIO()
with deflate.DeflateIO(stream, deflate.ZLIB) as d:
    d.write(uncompressed_data)
compressed_data = stream.getvalue()

Deflate window size¶

The window size limits how far back in the stream the (de)compressor can reference. Increasing the window size will improve compression, but will require more memory and make the compressor slower.

If an input stream was compressed a given window size, then DeflateIO using a smaller window size will fail mid-way during decompression with OSError, but only if a back-reference actually refers back further than the decompressor’s window size. This means it may be possible to decompress with a smaller window size. For example, this would trivially be the case if the original uncompressed data is shorter than the window size.

Decompression¶

The zlib format includes a header which specifies the window size that was used to compress the data. This indicates the maximum window size required to decompress this stream. If this header value is less than the specified wbits value (or if wbits is unset), then the header value will be used.

The gzip format does not include the window size in the header, and assumes that all gzip compressors (e.g. the gzip utility, or CPython’s implementation of gzip.GzipFile) use the maximum window size of 32kiB. For this reason, if the wbits parameter is not set, the decompressor will use a 32 kiB window size (corresponding to wbits set to 15). This means that to be able to decompress an arbitrary gzip stream, you must have at least this much RAM available. If you control the source data, consider instead using the zlib format with a smaller window size.

The raw format has no header and therefore does not include any information about the window size. If wbits is not set, then it will default to a window size of 256 bytes, which may not be large enough for a given stream. Therefore it is recommended that you should always explicitly set wbits if using the raw format.

Compression¶

For compression, MicroPython will default to a window size of 256 bytes for all formats. This provides a reasonable amount of compression with minimal memory usage and fast compression time, and will generate output that will work with any decompressor.

deflate — deflate compression & decompression¶