# End-to-End Optimization of Incoherent Imaging for Classification Under Detector-Limited Readout

arXiv: https://arxiv.org/abs/2606.09792 · HTML: https://mlsanity.com/paper/2606.09792/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)
- [Design of Task-Specific Optical Systems Using Broadband Diffractive Neural Networks](https://mlsanity.com/paper/1909.06553/) (2019)
- [An Optical Frontend for a Convolutional Neural Network](https://mlsanity.com/paper/1901.03661/) (2018)
- [Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means](https://mlsanity.com/paper/1401.4788/) (2014)
- [End-to-End Optimization of Metasurfaces for Imaging with Compressed Sensing](https://mlsanity.com/paper/2201.12348/) (2022)
- [Analysis and Optimization of Aperture Design in Computational Imaging](https://mlsanity.com/paper/1712.04541/) (2017)
- [Information-driven design of imaging systems](https://mlsanity.com/paper/2405.20559/) (2024)
- [Optimal Sensor Placement and Enhanced Sparsity for Classification](https://mlsanity.com/paper/1310.4217/) (2013)
- [Designing lensless imaging systems to maximize information capture](https://mlsanity.com/paper/2506.08513/) (2025)
- [Bounds on mutual information of mixture data for classification tasks](https://mlsanity.com/paper/2101.11670/) (2021)
- [End-to-end metasurface design for temperature imaging via broadband Planck-radiation regression](https://mlsanity.com/paper/2409.08456/) (2024)
- [Computationally Efficient Information-Driven Optical Design with Interchanging Optimization](https://mlsanity.com/paper/2507.07789/) (2025)

## Abstract (arXiv)

End-to-end co-optimization of optical front-ends (e.g. metasurfaces) and neural network back-ends has been widely applied to imaging tasks, yet a formalism characterizing when and why such systems outperform conventional lens-based imaging is largely lacking. This paper focuses on object classification, a central imaging task, and asks when end-to-end optimization of a phase mask for incoherent imaging improves performance over a conventional focusing lens. We find that these gains arise primarily under constrained detector readout and are limited under full detector readout. In the latter setting, we prove that no incoherent phase mask exceeds the ideal-channel mutual information between detector measurements and class labels; a conventional focusing lens approaches this ceiling, and joint optimization yields no empirical gain. When detector readout is constrained -- by coarse spatial sampling or a limited number of measurements -- optimized optics can substantially improve classification by increasing class separability in the detector measurements. These gains are largest under low detector noise and shrink as noise grows, because the optics shape the signal before it reaches the detector but cannot remove noise added afterward. The advantage also depends on the spectral structure of the task: co-design helps most when class-discriminative content is concentrated at lower spatial frequencies than within-class variation. We develop a theoretical framework formalizing these distinctions and test its predictions on synthetic data and standard benchmarks (MNIST, FashionMNIST, SVHN).