Multispectral Thermal (PAG7936)

The PAG7936 variant of the Multispectral Thermal Camera Module pairs a 1 MP global-shutter colour sensor with a FLIR Lepton thermal core, so the OpenMV Cam can run colour-vision and thermal pipelines side by side.

Multispectral Thermal (PAG7936)

For full datasheet, photos, and ordering see the Multispectral Thermal product page.

Note

Supported on the OpenMV N6 only.

Highlights

  • PAG7936: 1 MP global shutter

  • Accepts FLIR Lepton 1.x / 2.x / 3.x thermal cores

  • Simultaneous thermal + colour processing on one module

  • Sees in complete darkness, supports temperature measurement

  • Global shutter handles fast motion without rolling-shutter artifacts

Usage

The colour sensor and the FLIR Lepton each get their own csi.CSI instance. The first call defaults to the primary sensor (the PAG7936); the second binds to the Lepton by passing cid= csi.LEPTON. Hard-reset the colour sensor with csi.CSI.reset (hard=True) to bring the rail up, and configure the Lepton with hard=False so its driver only reprograms the chip without re-toggling reset.

csi.CSI.framesize ( csi.QVGA ) matches the Lepton output to the colour camera, so each snapshot() returns a 320x240 frame. The Lepton driver internally upscales its 80x60 (1.x/2.x) or 160x120 (3.x) native frame to the requested size — at QVGA every Lepton pixel covers a 4x4 or 2x2 cell on the colour frame.

Two scratch buffers stay constant across the frame loop — a 256x1 alpha palette stored as an image.Image so cool Lepton pixels become transparent and hot pixels become opaque (the quadratic ramp suppresses background detail without crushing the mid-range), and a Lepton frame buffer pre-allocated with image.Image so csi.CSI.snapshot (blocking=False, image=...) can fill it in place each iteration without reallocating:

import time
import csi
import image
import math

alpha_pal = image.Image(256, 1, image.GRAYSCALE)
for i in range(256):
    alpha_pal[i] = int(math.pow((i / 255), 2) * 255)

# Setup the color camera sensor.
csi0 = csi.CSI()
csi0.reset(hard=True)  # force hardware reset.
csi0.pixformat(csi.RGB565)
csi0.framesize(csi.QVGA)

csi1 = csi.CSI(cid=csi.LEPTON)
csi1.reset(hard=False)  # no hardware reset - just configure lepton
csi1.pixformat(csi.GRAYSCALE)
csi1.framesize(csi.QVGA)

# Optional temperature range controls for the LEPTON.
# csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, False)
# csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 20.0, 40.0)

clock = time.clock()

img1 = image.Image(csi1.width(), csi1.height(), csi1.pixformat())

while True:
    clock.tick()
    img0 = csi0.snapshot()
    csi1.snapshot(blocking=False, image=img1)
    img0.draw_image(img1, 0, 0, color_palette=image.PALETTE_IRONBOW,
                    alpha_palette=alpha_pal,
                    hint=image.BILINEAR)
    print(clock.fps())

Each iteration takes a blocking colour snapshot and a non-blocking Lepton snapshot — the Lepton runs at 9 Hz so blocking on it would throttle the colour pipeline. Image.draw_image then composites the two: color_palette= image.PALETTE_IRONBOW maps the Lepton’s grayscale to a FLIR-style warm colour ramp, alpha_palette= blends each pixel using the quadratic alpha map, and hint= image.BILINEAR smooths the upscale.

Temperature measurement

Radiometric Leptons (Lepton 2.5 / 3.5) report calibrated per-pixel temperature data. Enable measurement mode through csi.CSI.ioctl with csi.IOCTL_LEPTON_SET_MODE, then clamp the temperature window with csi.IOCTL_LEPTON_SET_RANGE (min_celsius, max_celsius). The Lepton driver linearly maps grayscale pixel value 0 to min_celsius and 255 to max_celsius, so each pixel becomes a temperature lookup within the configured window. Pixels colder than min_celsius saturate at 0, pixels hotter than max_celsius saturate at 255.

csi.IOCTL_LEPTON_SET_MODE takes two flags. The first turns measurement on; the second selects the sensor’s temperature range:

  • Low range — (True, False) — sensor span -10 °C to +140 °C (room-scale scenes). Clamp the window to the area of interest, e.g. (20.0, 40.0) for body-heat tracking:

    csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, False)
csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 20.0, 40.0)

  • High range — (True, True) — sensor span -10 °C to ~+450 °C typical (~+400 °C at room temperature) for hot objects. Clamp to e.g. (0.0, 400.0) for furnace or hot-element tracking:

    csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, True)
csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 0.0, 400.0)

To convert a grayscale pixel back to Celsius:

def p_to_temp(p, min_t, max_t):
    return (p * (max_t - min_t)) / 255.0 + min_t

This works on individual pixels or on aggregated statistics (e.g. stats.mean() from Image.get_statistics) inside an ROI when locating hot/cool regions with Image.find_blobs.

GPU-accelerated alignment

Image.draw_image accepts a transform= argument — a 3x3 homography matrix as a 2-D ulab.numpy array. On the OpenMV N6 the GPU runs the per-pixel projection during the same draw, so the Lepton frame can be re-aligned against the colour camera’s perspective without a separate warp pass. Calibrate the matrix per camera with the thermal-overlay-calibration tool:

import time
import csi
import image
from ulab import numpy as np
import math

# Calibration matrix from the thermal-overlay-calibration tool.
m = np.array([
    [3.704807, 0.257018, 37.260564],
    [0.052147, 3.609977, -7.831831],
    [0.000294, 0.000552, 1.000000],
])

alpha_pal = image.Image(256, 1, image.GRAYSCALE)
for i in range(256):
    alpha_pal[i] = int(math.pow((i / 255), 2) * 255)

# Setup the color camera sensor.
csi0 = csi.CSI()
csi0.reset(hard=True)  # force hardware reset.
csi0.pixformat(csi.RGB565)
csi0.framesize(csi.VGA)

csi1 = csi.CSI(cid=csi.LEPTON)
csi1.reset(hard=False)  # no hardware reset - just configure lepton
csi1.pixformat(csi.GRAYSCALE)
csi1.framesize(csi.QQVGA)

# Optional temperature range controls for the LEPTON.
# csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, False)
# csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 20.0, 40.0)

clock = time.clock()

img1 = image.Image(csi1.width(), csi1.height(), csi1.pixformat())

while True:
    clock.tick()
    img0 = csi0.snapshot()
    csi1.snapshot(blocking=False, image=img1)
    img0.draw_image(img1, 0, 0, color_palette=image.PALETTE_IRONBOW,
                    alpha_palette=alpha_pal,
                    hint=image.BILINEAR,
                    transform=m)
    print(clock.fps())

Note that this variant runs the colour camera at csi.VGA (640x480) and the Lepton at csi.QQVGA (160x120) — the homography projects the smaller Lepton frame into the larger colour frame as part of the draw, so the upscale factor is baked into the matrix itself rather than applied separately.