# Convolutional neural networks that teach microscopes how to image

arXiv: https://arxiv.org/abs/1709.07223 · HTML: https://mlsanity.com/paper/1709.07223/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2017.

## Built on by (most cited first, 13 of 13)

- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile Vision](https://mlsanity.com/paper/2109.08880/) (2021)
- [Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging](https://mlsanity.com/paper/2204.09871/) (2022)
- [PUERT: Probabilistic Under-sampling and Explicable Reconstruction Network for CS-MRI](https://mlsanity.com/paper/2204.11189/) (2022)
- [Increasing a microscope's effective field of view via overlapped imaging and machine learning](https://mlsanity.com/paper/2110.04921/) (2021)
- [The Berkeley Single Cell Computational Microscopy (BSCCM) Dataset](https://mlsanity.com/paper/2402.06191/) (2024)
- [Machine vision with small numbers of detected photons per inference](https://mlsanity.com/paper/2603.23974/) (2026)
- [Differentiable Microscopy Designs an All Optical Phase Retrieval Microscope](https://mlsanity.com/paper/2203.14944/) (2022)
- [Differentiable Microscopy for Content and Task Aware Compressive Fluorescence Imaging](https://mlsanity.com/paper/2203.14945/) (2022)
- [Collaborative On-Sensor Array Cameras](https://mlsanity.com/paper/2506.04061/) (2025)
- [Noise-Adaptive Intelligent Programmable Meta-Imager](https://mlsanity.com/paper/2208.10171/) (2022)
- [Aperture-aware Dispersion 5-D Light-field Imaging Spectrometer](https://mlsanity.com/paper/2607.04635/) (2026)

## Abstract (arXiv)

Deep learning algorithms offer a powerful means to automatically analyze the content of medical images. However, many biological samples of interest are primarily transparent to visible light and contain features that are difficult to resolve with a standard optical microscope. Here, we use a convolutional neural network (CNN) not only to classify images, but also to optimize the physical layout of the imaging device itself. We increase the classification accuracy of a microscope's recorded images by merging an optical model of image formation into the pipeline of a CNN. The resulting network simultaneously determines an ideal illumination arrangement to highlight important sample features during image acquisition, along with a set of convolutional weights to classify the detected images post-capture. We demonstrate our joint optimization technique with an experimental microscope configuration that automatically identifies malaria-infected cells with 5-10% higher accuracy than standard and alternative microscope lighting designs.