Prevent Frame Buffering with OpenCV

My software team’s current project involves doing frame capture from a set of cameras to identify the characteristics of products on a conveyor belt. We ran into a problem with frame buffering, and here’s what we did.

The team got around to prototyping the control logic for the system before the actual camera hardware was available to test and integrate with. We wanted to get a feel for actually integrating with a camera and solving some of the networking problems involved. So, we created a demo implementation of the “frame capturing interface” powered by OpenCV instead of the camera manufacturer’s SDK.

The first attempt was very basic and simply wrapped OpenCV behind an interface we could match later on with the hardware software development kit (SDK).

class BasicFrameCapture:
    def __init__(self, name):
        self.cap = cv2.VideoCapture(name)
        if not self.cap.isOpened():
            raise Exception("Could not open video.")

    def read(self) -> ImgType:
        ret, frame = self.cap.read()
        if not ret:
            raise Exception("Could not read frame.")
        return frame

Then, to give this capture something to read from that would behave somewhat like a real networked camera, we used VLC to serve a webcam’s feed over a raw HTTP stream:

cvlc v4l2:///dev/video0 \
  --sout '#transcode{vcodec="mp1v"}:standard{mux=raw,access=http,dst=":1234"}'

And, then, we can instantiate our capture class and start pulling frames from the stream with “http://localhost:1234” as the name param.

Problem

However, we immediately noticed a problem: our feed was lagging way behind reality. Grabbing the feed directly with VLC looked fine. It also looked fine if we called our capture class in an infinite loop that grabbed frames as fast as possible. Our control logic wasn’t running at top speed though. Instead, we were capturing once every few seconds on a trigger.

Our theory was that, under the hood, OpenCV was buffering up every frame sent by the HTTP stream. That meant we were seeing the frames come through one at a time, in order. Since the frames were coming into the queue at 24 frames per second, and we were only reading frames out at one every few seconds, our “view” of the feed lagged far behind real-time.

Fix

The fix was to decouple the reading of frames from the stream from the actual logical read, like so:

class BufferlesCvCapture:
    def __init__(self, name):
        self.name = name
        self.cap = cv2.VideoCapture(name)
        if not self.cap.isOpened():
            raise Exception("Could not open video.")
        self.q = queue.Queue()
        t = threading.Thread(target=self._reader)
        t.daemon = True
        t.start()

    # read frames as soon as they are available, keeping only most recent one
    def _reader(self):
        while True:
            ret, frame = self.cap.read()
            if not ret:
                raise Exception("Could not read frame.")
            if not self.q.empty():
                try:
                    self.q.get_nowait()  # discard previous (unprocessed) frame
                except queue.Empty:
                    pass
            self.q.put(frame)

    def read(self) -> ImgType:
        return self.q.get()

This class creates a thread that continuously reads frames out of the feed as quickly as they’re available and writes them into a queue. (This happens after discarding the previous frame if it wasn’t consumed.) The read method is then free to be called on whatever trigger the control logic needs and will always see a fresh, up-to-date frame.