# FourierNets enable the design of highly non-local optical encoders for computational imaging

arXiv: https://arxiv.org/abs/2104.10611 · HTML: https://mlsanity.com/paper/2104.10611/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network](https://mlsanity.com/paper/1703.02719/) (2017)
- [Fast Training of Convolutional Networks through FFTs](https://mlsanity.com/paper/1312.5851/) (2013)
- [Spectral Representations for Convolutional Neural Networks](https://mlsanity.com/paper/1506.03767/) (2015)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Unrolled Optimization with Deep Priors](https://mlsanity.com/paper/1705.08041/) (2017)
- [DWDN: Deep Wiener Deconvolution Network for Non-Blind Image Deblurring](https://mlsanity.com/paper/2103.09962/) (2021)
- [Learned reconstructions for practical mask-based lensless imaging](https://mlsanity.com/paper/1908.11502/) (2019)
- [Fast Convolutional Nets With fbfft: A GPU Performance Evaluation](https://mlsanity.com/paper/1412.7580/) (2014)
- [Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy](https://mlsanity.com/paper/2010.05382/) (2020)
- [Simultaneous whole-animal 3D-imaging of neuronal activity using light field microscopy](https://mlsanity.com/paper/1401.5333/) (2014)
- [Fourier DiffuserScope: Single-shot 3D Fourier light field microscopy with a diffuser](https://mlsanity.com/paper/2006.16343/) (2020)

## Built on by (most cited first, 3 of 3)

- [Generalized Compressed Sensing for Image Reconstruction with Diffusion Probabilistic Models](https://mlsanity.com/paper/2405.17456/) (2024)
- [Generalizing Deep Surrogate Solvers for Broadband Electromagnetic Field Prediction at Unseen Wavelengths](https://mlsanity.com/paper/2408.02971/) (2024)
- [From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy](https://mlsanity.com/paper/2510.22577/) (2025)

## Abstract (arXiv)

Differentiable simulations of optical systems can be combined with deep learning-based reconstruction networks to enable high performance computational imaging via end-to-end (E2E) optimization of both the optical encoder and the deep decoder. This has enabled imaging applications such as 3D localization microscopy, depth estimation, and lensless photography via the optimization of local optical encoders. More challenging computational imaging applications, such as 3D snapshot microscopy which compresses 3D volumes into single 2D images, require a highly non-local optical encoder. We show that existing deep network decoders have a locality bias which prevents the optimization of such highly non-local optical encoders. We address this with a decoder based on a shallow neural network architecture using global kernel Fourier convolutional neural networks (FourierNets). We show that FourierNets surpass existing deep network based decoders at reconstructing photographs captured by the highly non-local DiffuserCam optical encoder. Further, we show that FourierNets enable E2E optimization of highly non-local optical encoders for 3D snapshot microscopy. By combining FourierNets with a large-scale multi-GPU differentiable optical simulation, we are able to optimize non-local optical encoders 170× to 7372× larger than prior state of the art, and demonstrate the potential for ROI-type specific optical encoding with a programmable microscope.