What Can SPADs Do Alone?

In previous demos, we’ve shown the potential of SPAD sensors paired with Ubicept’s innovations, often highlighting their advantages over conventional sensors. As exciting as those results were, they also prompted an important question from many of our readers:

Impressive results, but how much of this can be attributed to the performance of the SPAD itself versus Ubicept processing?

In other words, could you achieve similar results by buying a commercially available SPAD camera (like the Canon MS-500) and using it out of the box? Or what about running the color SPAD camera from our own development kit with no additional processing?

To answer this, we’ll look more closely at the data from one of our most recent demos. In case you missed it, check it out below:

Raw SPAD data?

SPAD sensors capture the arrivals of individual photons with incredibly high temporal precision. This makes it tricky to show what a truly raw frame looks like. However, we can approximate it by counting the number of photon arrivals per pixel during a very short exposure window:

At first glance, these noisy frames look like what you’d get from a low-light conventional camera using a similarly short exposure window. You even might assume that SPADs don’t offer much of an advantage! We’ll explain later why this isn’t the case, but first, let’s talk about some common approaches to handling this noise.

Other approaches

One simple way to reduce noise is to gather more light by lengthening the exposure window. This works for both conventional and SPAD-based imaging. However, when motion is present, the outcome is similar:

But what about AI-based video upscaling and denoising tools? They’re widely available today, so it’s natural to wonder how they might perform here. We were curious ourselves, so we experimented with one of the more popular commercial options:

This looks less noisy than the input while preserving certain details, but areas with lower brightness or contrast show obvious deficiencies.

Ubicept Photon Fusion

Now, let’s see how Ubicept Photon Fusion performs on the same input:

Here’s a side-by-side comparison of all the results:

So, to answer the question that we started with: SPADs can provide incredibly rich raw data, but unlocking their potential requires advanced processing. AI-based video denoisers are one form of advanced processing that can help, but Ubicept Photon Fusion performs significantly better.

With that said, we’re not trying to claim that Ubicept Photon Fusion is universally better. It’s more that the two approaches make fundamentally different assumptions about the form of the input and purpose of the output. To elaborate:

1. Ubicept Photon Fusion is optimized for sensors with extreme time precision and specific noise properties. While both conventional sensors and SPADs experience shot noise, only conventional sensors introduce read noise. By leveraging this distinction, Ubicept Photon Fusion can achieve vastly superior results. In contrast, AI-based video denoisers take a general approach to handle many types of noise and other artifacts.
2. ‍Ubicept Photon Fusion doesn’t treat sensors as passive data providers. Instead, it works in tandem with FLARE (our Flexible Light Acquisition and Representation Engine) to optimize the capture process itself. AI-based video denoisers, on the other hand, are designed to process arbitrary footage that may have undergone prior modifications such as tone mapping, video compression, and even prior noise reduction passes.
3. ‍Ubicept Photon Fusion ultimately seeks to enhance perception, while AI-based video denoisers focus on visually pleasing results. What’s the difference? Well, because AI-based video denoisers rely on learned patterns, they risk generating details that look real, but aren’t. In contrast, Ubicept Photon Fusion preserves the precision of photon arrival data, ensuring that downstream perception systems receive reliable, physics-based information.

To put it another way: if you want to clean up a random smartphone video that you shot last year, commercial AI-based video denoising tools will likely do a better job than Ubicept Photon Fusion. But if your next smartphone has Ubicept Photon Fusion, it won’t just enhance your videos—it’ll support a fundamentally higher level of perception than what’s possible today.