# Image Denoising and the Generative Accumulation of Photons

arXiv: https://arxiv.org/abs/2307.06607 · HTML: https://mlsanity.com/paper/2307.06607/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [Noise2Void - Learning Denoising from Single Noisy Images](https://mlsanity.com/paper/1811.10980/) (2018)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [Noise2Self: Blind Denoising by Self-Supervision](https://mlsanity.com/paper/1901.11365/) (2019)
- [Soft Diffusion: Score Matching for General Corruptions](https://mlsanity.com/paper/2209.05442/) (2022)
- [A Poisson-Gaussian Denoising Dataset with Real Fluorescence Microscopy Images](https://mlsanity.com/paper/1812.10366/) (2018)
- [Interpretable Unsupervised Diversity Denoising and Artefact Removal](https://mlsanity.com/paper/2104.01374/) (2021)
- [N2V2 -- Fixing Noise2Void Checkerboard Artifacts with Modified Sampling Strategies and a Tweaked Network Architecture](https://mlsanity.com/paper/2211.08512/) (2022)

## Built on by (most cited first, 3 of 3)

- [Generalized Recorrupted-to-Recorrupted: Self-Supervised Learning Beyond Gaussian Noise](https://mlsanity.com/paper/2412.04648/) (2024)
- [bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction](https://mlsanity.com/paper/2410.23247/) (2024)
- [Unsupervised Denoising for Signal-Dependent and Row-Correlated Imaging Noise](https://mlsanity.com/paper/2310.07887/) (2023)

## Abstract (arXiv)

We present a fresh perspective on shot noise corrupted images and noise removal. By viewing image formation as the sequential accumulation of photons on a detector grid, we show that a network trained to predict where the next photon could arrive is in fact solving the minimum mean square error (MMSE) denoising task. This new perspective allows us to make three contributions: We present a new strategy for self-supervised denoising, We present a new method for sampling from the posterior of possible solutions by iteratively sampling and adding small numbers of photons to the image. We derive a full generative model by starting this process from an empty canvas. We call this approach generative accumulation of photons (GAP). We evaluate our method quantitatively and qualitatively on 4 new fluorescence microscopy datasets, which will be made available to the community. We find that it outperforms supervised, self-supervised and unsupervised baselines or performs on-par.