MicroPython manifest files

Summary

MicroPython has a feature that allows Python code to be “frozen” into the firmware, as an alternative to loading code from the filesystem.

This has the following benefits:

  • the code is pre-compiled to bytecode, avoiding the need for the Python source to be compiled at load-time.

  • the bytecode can be executed directly from ROM (i.e. flash memory) rather than being copied into RAM. Similarly any constant objects (strings, tuples, etc) are loaded from ROM also. This can lead to significantly more memory being available for your application.

  • on devices that do not have a filesystem, this is the only way to load Python code.

During development, freezing is generally not recommended as it will significantly slow down your development cycle, as each update will require re-flashing the entire firmware. However, it can still be useful to selectively freeze some rarely-changing dependencies (such as third-party libraries).

The way to list the Python files to be frozen into the firmware is via a “manifest”, which is a Python file that will be interpreted by the build process. Typically you would write a manifest file as part of a board definition, but you can also write a stand-alone manifest file and use it with an existing board definition.

Manifest files can define dependencies on libraries from micropython-lib as well as Python files on the filesystem, and also on other manifest files.

Writing manifest files

A manifest file is a Python file containing a series of function calls. See the available functions defined below.

Any paths used in manifest files can include the following variables. These all resolve to absolute paths.

  • $(MPY_DIR) – path to the micropython repo.

  • $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to use require().

  • $(PORT_DIR) – path to the current port (e.g. ports/stm32)

  • $(BOARD_DIR) – path to the current board (e.g. ports/stm32/boards/OPENMV4)

Custom manifest files should not live in the main MicroPython repository. You should keep them in version control with the rest of your project.

Typically a manifest used for compiling firmware will need to include the port manifest, which might include frozen modules that are required for the board to function. If you just want to add additional modules to an existing board, then include the board manifest (which will in turn include the port manifest).

Building with a custom manifest

Your manifest can be specified on the make command line with:

$ make BOARD=MYBOARD FROZEN_MANIFEST=/path/to/my/project/manifest.py

This applies to all ports, including CMake-based ones (e.g. rp2), as the Makefile wrapper will pass this into the CMake build.

Adding a manifest to a board definition

If you have a custom board definition, you can make it include your custom manifest automatically. On make-based ports (most ports), in your mpconfigboard.mk set the FROZEN_MANIFEST variable.

FROZEN_MANIFEST ?= $(BOARD_DIR)/manifest.py

On CMake-based ports (e.g. rp2), instead use mpconfigboard.cmake

set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py)

High-level functions

These are the functions you will normally use. They add code to the set that is precompiled to bytecode and frozen into the firmware image:

  • module and package freeze your own local source — a single file or a whole package directory respectively.

  • require freezes a published package (and its dependencies) from micropython-lib, by name.

  • include pulls in another manifest so its frozen modules are added too.

  • add_library and metadata are support functions (registering extra search paths for require, and declaring package metadata).

A typical firmware manifest first includes the port or board manifest (so the modules the board needs stay frozen), then adds its own module/package/require lines.

Note: The opt keyword argument can be set on the various functions, this controls the optimisation level used by the cross-compiler. See micropython.opt_level().

add_library(library, library_path, prepend=False)

Register the path to an external named library.

Use this when you want require to resolve packages from a directory other than micropython-lib — for example your own collection of drivers, or a third-party library checkout.

The path library_path will be automatically searched when using require. By default the added library is added to the end of the list of libraries to search. Pass True to prepend to add it to the start of the list.

Additionally, the added library can be explicitly requested by using require("name", library="library").

package(package_path, files=None, base_path='.', opt=None)

Freeze an entire package — a directory of .py files (optionally with sub-packages) — so it can be imported as import <package>. Use module instead for a single standalone file.

This is equivalent to copying the “package_path” directory to the device (except as frozen code).

In the simplest case, to freeze a package “foo” in the current directory:

package("foo")

will recursively include all .py files in foo, and will be frozen as foo/**/*.py.

If the package isn’t in the same directory as the manifest file, use base_path:

package("foo", base_path="path/to/libraries")

You can use the variables above, such as $(PORT_DIR) in base_path.

To restrict to certain files in the package use files (note: paths should be relative to the package): package("foo", files=["bar/baz.py"]).

module(module_path, base_path='.', opt=None)

Freeze a single standalone .py file so it can be imported by its name (module("foo.py") makes import foo work). Use package for a directory/package.

If the file is in the current directory:

module("foo.py")

Otherwise use base_path to locate the file:

module("foo.py", base_path="src/drivers")

You can use the variables above, such as $(PORT_DIR) in base_path.

require(name, library=None)

Require a package by name (and its dependencies) from micropython-lib.

This is how standard-library extensions and community drivers get frozen in: the named package is fetched from the micropython-lib submodule and frozen along with everything it depends on. Use module or package instead to freeze your own source rather than a published package.

Optionally specify library (a string) to reference a package from a library that has been previously registered with add_library. Otherwise the list of library paths will be used.

include(manifest_path)

Include another manifest. This is how manifests are composed: a custom firmware manifest should include the port (or board) manifest so the modules the board needs stay frozen, and then add its own entries.

Typically a manifest used for compiling firmware will need to include the port manifest, which might include frozen modules that are required for the board to function.

The manifest argument can be a string (filename) or an iterable of strings.

Relative paths are resolved with respect to the current manifest file.

If the path is to a directory, then it implicitly includes the manifest.py file inside that directory.

You can use the variables above, such as $(PORT_DIR) in manifest_path.

metadata(description=None, version=None, license=None, author=None)

Define metadata for this manifest file. This is useful for manifests for micropython-lib packages.

These fields are consumed when a package is published to / installed from micropython-lib via mip; they are not needed in a board firmware manifest.

Low-level functions

These functions are documented for completeness, but with the exception of freeze_as_str all functionality can be accessed via the high-level functions.

The freeze* functions differ only in how the code is stored:

  • freeze_as_mpy / freeze_mpy store precompiled bytecode (.mpy) in flash. The code runs directly from flash, uses minimal RAM, and imports quickly. This is what module, package and require use internally.

  • freeze_as_str instead freezes the Python source, which is compiled to bytecode at import time (using RAM, and requiring the on-device compiler). This is the one capability not exposed by the high-level functions, which is why it is the exception noted above.

freeze(path, script=None, opt=0)

The underlying primitive the high-level functions build on; prefer those. Freeze the input specified by path, automatically determining its type. A .py script will be compiled to a .mpy first then frozen, and a .mpy file will be frozen directly.

path must be a directory, which is the base directory to begin searching for files. When importing the resulting frozen modules, the name of the module will start after path, i.e. path is excluded from the module name.

If path is relative, it is resolved to the current manifest.py.

If script is None, all files in path will be frozen.

If script is an iterable then freeze() is called on all items of the iterable (with the same path and opt passed through).

If script is a string then it specifies the file or directory to freeze, and can include extra directories before the file or last directory. The file or directory will be searched for in path. If script is a directory then all files in that directory will be frozen.

opt is the optimisation level to pass to mpy-cross when compiling .py to .mpy. These levels are described in micropython.opt_level().

freeze_as_str(path)

Freeze the given path and all .py scripts within it as a string, which will be compiled upon import. Use this only when the frozen code must remain Python source; it costs import-time RAM compared with the .mpy variants.

freeze_as_mpy(path, script=None, opt=0)

Freeze the input by first compiling the .py scripts to .mpy files, then freezing the resulting .mpy files. This is what module and package do under the hood. See freeze() for further details on the arguments.

freeze_mpy(path, script=None, opt=0)

Freeze the input, which must be .mpy files that are frozen directly (no compilation step). See freeze() for further details on the arguments.

Examples

To freeze a single file from the current directory which will be available as import mydriver, use:

module("mydriver.py")

To freeze a directory of files in a subdirectory “mydriver” of the current directory which will be available as import mydriver, use:

package("mydriver")

To freeze the “hmac” library from micropython-lib, use:

require("hmac")

A more complete example of a custom manifest.py file (for a board that has its own default manifest) is:

# Include the board's default manifest.
include("$(BOARD_DIR)/manifest.py")
# Add a custom driver
module("mydriver.py")
# Add aiorepl from micropython-lib
require("aiorepl")

Then the board can be compiled with

$ cd ports/stm32
$ make BOARD=MYBOARD FROZEN_MANIFEST=~/src/myproject/manifest.py

Note that most boards do not have their own manifest.py, rather they use the port one directly, in which case your manifest should just include("$(PORT_DIR)/boards/manifest.py") instead.