Motion Compensation for Conventional Sensors
Using Ubicept technology with off-the-shelf machine vision cameras
We just got back from our second year exhibiting at CES and it was intense! Among the questions that were asked at our booth, one of them came up quite frequently: “does your solution work with the cameras we’re using right now?” It’s a great question, but the answer is complicated. Let’s delve into it by starting with some background.
Context
Ubicept's core technology, invented by our co-founders at MIT and UW–Madison, was originally designed for Single Photon Avalanche Diode (SPAD) sensors. Unlike the conventional CMOS image sensors found in most cameras, which measure light intensity over a set exposure time, SPAD sensors count individual photons.
This may sound like exotic future technology, but it’s already in use today. Lower-resolution SPAD sensors have been a part of smartphones for 3D applications for some time. You probably have one in your pocket right now! Higher-resolution SPAD sensors exist too, but their cost currently makes them impractical for many applications.
To address the main question, we considered the key properties of SPAD sensors that we rely on to achieve our results:
Low Noise: Unlike conventional CMOS sensors, SPAD sensors don’t have read noise since they skip the analog-to-digital conversion step. We say “low noise” here and not “zero noise” because they still encounter other types of noise.
Precise Timing: SPAD sensors can detect photons arriving with picosecond precision. We leverage this in our custom firmware to “encode” photon frames that our software stack can “decode” into video frames with low noise and minimal motion blur.
The second property is why we can’t just clear up noisy and/or blurry footage from, say, a standard automotive or surveillance camera shooting at 30 fps. But this exploration made us wonder whether there were other conventional CMOS sensors that could get us closer to SPAD-like precision.
Experiment
We set a tight budget for this experiment which excluded most “affordable” high-speed cameras such as these. After all, one of the main reasons that higher-resolution SPAD cameras aren’t more prevalent today, despite their remarkable capabilities, is that they tend to be quite expensive. While their prices are expected to drop dramatically over the next few years, many of our prospective customers are looking for economical solutions sooner than that.
Our eventual choice was the PHX004S from LUCID Vision Labs. Let’s just say that our setup cost us a few hundred bucks including a C-mount lens and PoE adapter—we’ll respect their “login to see prices” button and not provide a specific number. This camera features a 0.4 MP Sony IMX287 sensor, which may not sound great by today’s standards, but it offers key advantages such as its low noise, high frame rate, global shutter, and programmable exposure controls.
For comparison, we decided to put it up against the main camera on an iPhone 15 Pro. Here are the specs, side by side:
| LUCID Vision Labs PHX004S | iPhone 15 Pro Main Camera | |
|---|---|---|
| Release Date | 2018 | 2023 |
| Max Resolution | 0.4 MP | 48 MP |
| Max Frame Rate | 291 fps | 240 fps |
| Shutter Type | Global | Rolling |
| Sensor Size | 5.0 x 3.8 mm | 9.8 x 7.3 mm |
| Image Stabilization | None | Sensor Shift |
This may not seem like a fair comparison given the 120x resolution and 4x sensor size advantage, but we wanted to choose a camera that readers of this blog would be familiar with.
Let’s start with a still photograph from on the iPhone, taken with a 0.5 second exposure in low light:
It’s obvious that no 0.4 MP camera can come close to this level of quality. The static parts of the image look incredibly clear. The issue here is that our subject, Wacky Waving Inflatable Tube Guy, is blurred beyond recognition due to his vigorous and undulating gesticulations. Let’s try the 120 fps “slo-mo” mode, with levels boosted in post processing to make details visible:
You can see the motion here, but the frames are practically unusable due to noise and compression artifacts. Lowering the frame rate to 24 fps strikes the best balance between noise and blur:
Now, let’s take a look at a 3 ms exposure from the IMX287-based PHX004S, with the gain cranked up all the way. To keep the comparison fair, we set our lens to f/1.8 to match the maximum aperture of the iPhone 15 Pro’s main camera:
There’s an extreme level of noise here—enough to make the texture of the carpet and the tape “X” very difficult to see—but the motion of Wacky Waving Inflatable Tube Guy is frozen quite well. So, let’s see what happens when we apply Ubicept processing to the input data:
As you can see, the noise is drastically reduced while revealing details in the motion and texture. We think it’s fair to say that these results look a lot better that what we were able to achieve with the iPhone! Check out the video below for the full results (apologies for the minor compression artifacts in the final results; it’s an side effect of using YouTube and we’d be happy to send you the uncompressed version if you want to pixel-peep):
Summary
So, to answer the question posed in the introduction: yes, our technology can work with economical cameras which are available today, but its effectiveness will depend on the capabilities of the camera. We remain committed to developing solutions that exploit the incredible capabilities of SPAD sensors, but we’re open to work with partners who want to evaluate our technology on other sensors as well.
If you’d like to learn more or discuss an evaluation, let us know!