15.8. Streaming frames

The most common real use of a custom channel is streaming image frames from the cam to a host program at the cam’s frame rate. The mechanics are subtler than they look: a JPEG can run to 25 KB or more, so the host reads it as several fragments, and the cam’s capture loop must be prevented from overwriting the buffer mid-read. The right pattern – shown here and used by the tools in openmv-projects/tools/ – latches the buffer until the host finishes pulling the last byte.

15.8.1. The cam side

A frame channel that captures into a single framebuffer, latches it on the host’s first read, and only takes the next snapshot once the host has consumed the full image:

import csi
import protocol

csi0 = csi.CSI()
csi0.reset()
csi0.pixformat(csi.RGB565)
csi0.framesize(csi.QVGA)
csi0.framebuffers(1)

img = csi0.snapshot()
img.compress(quality=85)
img_mv = memoryview(img.bytearray())
img_size = len(img_mv)
frame_available = True


class FrameChannel:
    def poll(self):
        return frame_available

    def size(self):
        return img_size

    def readp(self, offset, size):
        global frame_available
        end = offset + size
        mv = img_mv[offset:end]
        if end == img_size:
            # Host has just read the last byte of this frame --
            # release the buffer so the capture loop can refresh.
            frame_available = False
        return mv


ch = protocol.register(name='frame', backend=FrameChannel())

while True:
    if not frame_available:
        img = csi0.snapshot()
        img.compress(quality=85)
        img_mv = memoryview(img.bytearray())
        img_size = len(img_mv)
        frame_available = True
        ch.send_event(0x01)   # notify host that a new frame is ready

Four pieces are doing real work here:

  • frame_available is the latch. The capture loop only takes a new snapshot when it is False – meaning the host has pulled the last byte of the previous frame. The host’s read sets it back to False from inside readp once the final offset has been served. Without this guard, the next csi0.snapshot() would overwrite the buffer mid-read and the host would receive a frame stitched together from two captures.

  • readp rather than read is what the backend implements. The protocol library treats the returned buffer as authoritative and reads its bytes directly into the outgoing packet – no copy. For frame-sized payloads readp is noticeably faster than read, which forces an intermediate copy.

  • size returns the cached JPEG length without recomputing anything; the capture loop maintains it whenever it refreshes the buffer. The host calls size between poll and readp to know how many bytes to pull.

  • send_event() notifies the host the instant a new frame lands so it can begin pulling without polling. The event ID 0x01 is application-defined (“frame ready” in this case); use a different small integer for each kind of notification.

15.8.2. Fragmentation

QVGA RGB565 at JPEG quality 85 compresses to roughly 10-25 KB, depending on the scene – much bigger than the negotiated max payload on any cam (see the per-board table in protocol.init()). One JPEG read won’t fit in one packet, and that’s fine, because the protocol library fragments it transparently.

When the host asks for channel_read('frame', 12000):

  1. The cam’s readp is called once with offset=0 and the full 12000-byte request. It returns one memoryview covering the whole range.

  2. The protocol library breaks that memoryview into max-payload- sized fragments on the wire, one CHANNEL_READ reply packet per fragment, each with its own header and CRC. The bytes are streamed out of the backend’s buffer directly – no copy.

  3. The host receives the fragments in order, the reliability layer retransmits any one chunk that fails its CRC, and the host SDK glues the chunks into the 12000-byte result returned to the caller.

Note

This is the key practical difference between readp and read. readp is called once per host request; the protocol layer fragments and transmits out of the single returned buffer. read is called once per fragment, and the library copies each returned chunk into its own packet buffer. For frame-sized payloads readp saves both the per-fragment Python-level call overhead and the copy.

Tip

Want to see the gap for yourself? Rename the backend’s readp method to read – nothing else changes; the library will pick up the read capability instead – and compare the host’s frame-rate counter before and after. The slower number is the per-fragment copy and Python-call cost you avoid by using readp.

The latch in FrameChannel.readp releases the buffer when offset + size == img_size – the moment the host has pulled the last byte. Until then, the buffer must stay valid, which is why the capture loop only takes the next snapshot once frame_available flips back to False.

15.8.3. The host side

The host pulls frames in a tight loop:

import io
from PIL import Image
from openmv.camera import Camera

with Camera('/dev/ttyACM0', baudrate=921600) as cam:
    cam.update_channels()

    while True:
        size = cam.channel_size('frame')
        if not size:
            continue
        data = cam.channel_read('frame', size)
        img = Image.open(io.BytesIO(data))
        img.show()                  # or feed to a GUI

The channel_size() call doubles as a “is anything ready” check – zero means the cam hasn’t captured yet – so the loop skips read attempts on an empty buffer. For GUI applications that already poll on a timer, this is the natural pattern.

Pillow’s Image.open decodes the JPEG; the cam already JPEG-compressed it so the host doesn’t have to redo expensive bit-packing on RGB565. The host script could just as easily save the bytes to disk, hand them to OpenCV, or push them through a web view.

15.8.4. Throughput thinking

Three things bound the achievable frame rate:

  • The cam’s capture rate. The protocol can’t deliver frames faster than the sensor produces them; whatever cap the chosen pixel format and frame size impose on capture is the ceiling.

  • The negotiated max payload. Bigger payloads mean fewer fragments per frame and less framing overhead, so cams with larger protocol buffers move bytes faster than smaller ones.

  • CRC and ACK overhead. Each packet costs 14 bytes of framing plus one ACK round-trip. For long fragments the per-payload overhead is small; for tiny payloads it dominates.

For most cam-to-laptop GUI work the limiting factor is the cam’s capture and JPEG compression time, not the protocol stack. Where the protocol does become the bottleneck – streaming uncompressed raw frames at high frame rates, for example – the levers are turning off ACKs (protocol.init(ack=False)), increasing the protocol buffer if the cam supports it, or capturing in GRAYSCALE so each compressed JPEG carries one channel instead of three and the encoded frame ends up noticeably smaller on the wire.

The frame channel is the canonical cam-to-host data flow. The same backend interface, with a write method added, lets the host push data the other way too – which is what an interactive cam tool needs as soon as the operator wants to change something rather than just watch.