# Differentiable probabilistic models of scientific imaging with the Fourier slice theorem

arXiv: https://arxiv.org/abs/1906.07582 · HTML: https://mlsanity.com/paper/1906.07582/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 6 of 6)

- [CryoAI: Amortized Inference of Poses for Ab Initio Reconstruction of 3D Molecular Volumes from Real Cryo-EM Images](https://mlsanity.com/paper/2203.08138/) (2022)
- [Deep Generative Modeling for Volume Reconstruction in Cryo-Electron Microscopy](https://mlsanity.com/paper/2201.02867/) (2022)
- [End-to-End Simultaneous Learning of Single-particle Orientation and 3D Map Reconstruction from Cryo-electron Microscopy Data](https://mlsanity.com/paper/2107.02958/) (2021)
- [Boosted ab initio Cryo-EM 3D Reconstruction with ACE-EM](https://mlsanity.com/paper/2302.06091/) (2023)
- [Improved Cryo-EM Pose Estimation and 3D Classification through Latent-Space Disentanglement](https://mlsanity.com/paper/2308.04956/) (2023)
- [CryoChains: Heterogeneous Reconstruction of Molecular Assembly of Semi-flexible Chains from Cryo-EM Images](https://mlsanity.com/paper/2306.07274/) (2023)

## Abstract (arXiv)

Scientific imaging techniques such as optical and electron microscopy and computed tomography (CT) scanning are used to study the 3D structure of an object through 2D observations. These observations are related to the original 3D object through orthogonal integral projections. For common 3D reconstruction algorithms, computational efficiency requires the modeling of the 3D structures to take place in Fourier space by applying the Fourier slice theorem. At present, it is unclear how to differentiate through the projection operator, and hence current learning algorithms can not rely on gradient based methods to optimize 3D structure models. In this paper we show how back-propagation through the projection operator in Fourier space can be achieved. We demonstrate the validity of the approach with experiments on 3D reconstruction of proteins. We further extend our approach to learning probabilistic models of 3D objects. This allows us to predict regions of low sampling rates or estimate noise. A higher sample efficiency can be reached by utilizing the learned uncertainties of the 3D structure as an unsupervised estimate of the model fit. Finally, we demonstrate how the reconstruction algorithm can be extended with an amortized inference scheme on unknown attributes such as object pose. Through empirical studies we show that joint inference of the 3D structure and the object pose becomes more difficult when the ground truth object contains more symmetries. Due to the presence of for instance (approximate) rotational symmetries, the pose estimation can easily get stuck in local optima, inhibiting a fine-grained high-quality estimate of the 3D structure.