# Clean Implicit 3D Structure from Noisy 2D STEM Images

arXiv: https://arxiv.org/abs/2203.15434 · HTML: https://mlsanity.com/paper/2203.15434/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [Noise2Void - Learning Denoising from Single Noisy Images](https://mlsanity.com/paper/1811.10980/) (2018)
- [Noise2Self: Blind Denoising by Self-Supervision](https://mlsanity.com/paper/1901.11365/) (2019)
- [Learned Primal-dual Reconstruction](https://mlsanity.com/paper/1707.06474/) (2017)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [Image reconstruction by domain transform manifold learning](https://mlsanity.com/paper/1704.08841/) (2017)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Escaping Plato's Cave: 3D Shape From Adversarial Rendering](https://mlsanity.com/paper/1811.11606/) (2018)
- [Low Dose CT Image Denoising Using a Generative Adversarial Network with Wasserstein Distance and Perceptual Loss](https://mlsanity.com/paper/1708.00961/) (2017)
- [Noise Flow: Noise Modeling with Conditional Normalizing Flows](https://mlsanity.com/paper/1908.08453/) (2019)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations](https://mlsanity.com/paper/2008.01639/) (2020)
- [A Poisson-Gaussian Denoising Dataset with Real Fluorescence Microscopy Images](https://mlsanity.com/paper/1812.10366/) (2018)
- [Computed Tomography Reconstruction Using Deep Image Prior and Learned Reconstruction Methods](https://mlsanity.com/paper/2003.04989/) (2020)
- [Radon Inversion via Deep Learning](https://mlsanity.com/paper/1808.03015/) (2018)
- [Lose The Views: Limited Angle CT Reconstruction via Implicit Sinogram Completion](https://mlsanity.com/paper/1711.10388/) (2017)
- [TomoGAN: Low-Dose Synchrotron X-Ray Tomography with Generative Adversarial Networks](https://mlsanity.com/paper/1902.07582/) (2019)
- [Microscopy Image Restoration with Deep Wiener-Kolmogorov filters](https://mlsanity.com/paper/1911.10989/) (2019)
- [Inviwo -- A Visualization System with Usage Abstraction Levels](https://mlsanity.com/paper/1811.12517/) (2018)

## Built on by (most cited first, 3 of 3)

- [High-resolution tomographic reconstruction of optical absorbance through scattering media using neural fields](https://mlsanity.com/paper/2304.01682/) (2023)
- [Differentiable Electron Microscopy Simulation: Methods and Applications for Visualization](https://mlsanity.com/paper/2205.04464/) (2022)
- [CellINR: Implicitly Overcoming Photo-induced Artifacts in 4D Live Fluorescence Microscopy](https://mlsanity.com/paper/2508.19300/) (2025)

## Abstract (arXiv)

Scanning Transmission Electron Microscopes (STEMs) acquire 2D images of a 3D sample on the scale of individual cell components. Unfortunately, these 2D images can be too noisy to be fused into a useful 3D structure and facilitating good denoisers is challenging due to the lack of clean-noisy pairs. Additionally, representing a detailed 3D structure can be difficult even for clean data when using regular 3D grids. Addressing these two limitations, we suggest a differentiable image formation model for STEM, allowing to learn a joint model of 2D sensor noise in STEM together with an implicit 3D model. We show, that the combination of these models are able to successfully disentangle 3D signal and noise without supervision and outperform at the same time several baselines on synthetic and real data.