Multispectral Thermal (PAG7936) =============================== The PAG7936 variant of the Multispectral Thermal Camera Module pairs a 1 MP global-shutter colour sensor with a FLIR Lepton thermal core, so the OpenMV Cam can run colour-vision and thermal pipelines side by side. .. image:: ../pag7936-hero.jpg :alt: Multispectral Thermal (PAG7936) :width: 400px :align: center For full datasheet, photos, and ordering see the `Multispectral Thermal product page `_. .. note:: Supported on the OpenMV N6 only. Highlights ---------- * PAG7936: 1 MP global shutter * Accepts FLIR Lepton 1.x / 2.x / 3.x thermal cores * Simultaneous thermal + colour processing on one module * Sees in complete darkness, supports temperature measurement * Global shutter handles fast motion without rolling-shutter artifacts Usage ----- The colour sensor and the FLIR Lepton each get their own `csi.CSI` instance. The first call defaults to the primary sensor (the PAG7936); the second binds to the Lepton by passing ``cid=`` `csi.LEPTON`. Hard-reset the colour sensor with `csi.CSI.reset` ``(hard=True)`` to bring the rail up, and configure the Lepton with ``hard=False`` so its driver only reprograms the chip without re-toggling reset. `csi.CSI.framesize` ``(`` `csi.QVGA` ``)`` matches the Lepton output to the colour camera, so each ``snapshot()`` returns a 320x240 frame. The Lepton driver internally upscales its 80x60 (1.x/2.x) or 160x120 (3.x) native frame to the requested size — at QVGA every Lepton pixel covers a 4x4 or 2x2 cell on the colour frame. Two scratch buffers stay constant across the frame loop — a 256x1 alpha palette stored as an `image.Image` so cool Lepton pixels become transparent and hot pixels become opaque (the quadratic ramp suppresses background detail without crushing the mid-range), and a Lepton frame buffer pre-allocated with `image.Image` so `csi.CSI.snapshot` ``(blocking=False, image=...)`` can fill it in place each iteration without reallocating:: import time import csi import image import math alpha_pal = image.Image(256, 1, image.GRAYSCALE) for i in range(256): alpha_pal[i] = int(math.pow((i / 255), 2) * 255) # Setup the color camera sensor. csi0 = csi.CSI() csi0.reset(hard=True) # force hardware reset. csi0.pixformat(csi.RGB565) csi0.framesize(csi.QVGA) csi1 = csi.CSI(cid=csi.LEPTON) csi1.reset(hard=False) # no hardware reset - just configure lepton csi1.pixformat(csi.GRAYSCALE) csi1.framesize(csi.QVGA) # Optional temperature range controls for the LEPTON. # csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, False) # csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 20.0, 40.0) clock = time.clock() img1 = image.Image(csi1.width(), csi1.height(), csi1.pixformat()) while True: clock.tick() img0 = csi0.snapshot() csi1.snapshot(blocking=False, image=img1) img0.draw_image(img1, 0, 0, color_palette=image.PALETTE_IRONBOW, alpha_palette=alpha_pal, hint=image.BILINEAR) print(clock.fps()) Each iteration takes a blocking colour snapshot and a non-blocking Lepton snapshot — the Lepton runs at 9 Hz so blocking on it would throttle the colour pipeline. `Image.draw_image` then composites the two: ``color_palette=`` `image.PALETTE_IRONBOW` maps the Lepton's grayscale to a FLIR-style warm colour ramp, ``alpha_palette=`` blends each pixel using the quadratic alpha map, and ``hint=`` `image.BILINEAR` smooths the upscale. Temperature measurement ~~~~~~~~~~~~~~~~~~~~~~~ Radiometric Leptons (Lepton 2.5 / 3.5) report calibrated per-pixel temperature data. Enable measurement mode through `csi.CSI.ioctl` with `csi.IOCTL_LEPTON_SET_MODE`, then clamp the temperature window with `csi.IOCTL_LEPTON_SET_RANGE` ``(min_celsius, max_celsius)``. The Lepton driver linearly maps grayscale pixel value 0 to ``min_celsius`` and 255 to ``max_celsius``, so each pixel becomes a temperature lookup within the configured window. Pixels colder than ``min_celsius`` saturate at 0, pixels hotter than ``max_celsius`` saturate at 255. `csi.IOCTL_LEPTON_SET_MODE` takes two flags. The first turns measurement on; the second selects the sensor's temperature range: * **Low range** — ``(True, False)`` — sensor span ``-10 °C`` to ``+140 °C`` (room-scale scenes). Clamp the window to the area of interest, e.g. ``(20.0, 40.0)`` for body-heat tracking:: csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, False) csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 20.0, 40.0) * **High range** — ``(True, True)`` — sensor span ``-10 °C`` to ``~+450 °C`` typical (``~+400 °C`` at room temperature) for hot objects. Clamp to e.g. ``(0.0, 400.0)`` for furnace or hot-element tracking:: csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, True) csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 0.0, 400.0) To convert a grayscale pixel back to Celsius:: def p_to_temp(p, min_t, max_t): return (p * (max_t - min_t)) / 255.0 + min_t This works on individual pixels or on aggregated statistics (e.g. ``stats.mean()`` from `Image.get_statistics`) inside an ROI when locating hot/cool regions with `Image.find_blobs`. GPU-accelerated alignment ~~~~~~~~~~~~~~~~~~~~~~~~~ `Image.draw_image` accepts a ``transform=`` argument — a 3x3 homography matrix as a 2-D ulab.numpy array. On the OpenMV N6 the GPU runs the per-pixel projection during the same draw, so the Lepton frame can be re-aligned against the colour camera's perspective without a separate warp pass. Calibrate the matrix per camera with the `thermal-overlay-calibration tool `_:: import time import csi import image from ulab import numpy as np import math # Calibration matrix from the thermal-overlay-calibration tool. m = np.array([ [3.704807, 0.257018, 37.260564], [0.052147, 3.609977, -7.831831], [0.000294, 0.000552, 1.000000], ]) alpha_pal = image.Image(256, 1, image.GRAYSCALE) for i in range(256): alpha_pal[i] = int(math.pow((i / 255), 2) * 255) # Setup the color camera sensor. csi0 = csi.CSI() csi0.reset(hard=True) # force hardware reset. csi0.pixformat(csi.RGB565) csi0.framesize(csi.VGA) csi1 = csi.CSI(cid=csi.LEPTON) csi1.reset(hard=False) # no hardware reset - just configure lepton csi1.pixformat(csi.GRAYSCALE) csi1.framesize(csi.QQVGA) # Optional temperature range controls for the LEPTON. # csi1.ioctl(csi.IOCTL_LEPTON_SET_MODE, True, False) # csi1.ioctl(csi.IOCTL_LEPTON_SET_RANGE, 20.0, 40.0) clock = time.clock() img1 = image.Image(csi1.width(), csi1.height(), csi1.pixformat()) while True: clock.tick() img0 = csi0.snapshot() csi1.snapshot(blocking=False, image=img1) img0.draw_image(img1, 0, 0, color_palette=image.PALETTE_IRONBOW, alpha_palette=alpha_pal, hint=image.BILINEAR, transform=m) print(clock.fps()) Note that this variant runs the colour camera at `csi.VGA` (640x480) and the Lepton at `csi.QQVGA` (160x120) — the homography projects the smaller Lepton frame into the larger colour frame as part of the draw, so the upscale factor is baked into the matrix itself rather than applied separately.