# Differentiable Microscopy for Content and Task Aware Compressive Fluorescence Imaging

arXiv: https://arxiv.org/abs/2203.14945 · HTML: https://mlsanity.com/paper/2203.14945/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Sketch-based Manga Retrieval using Manga109 Dataset](https://mlsanity.com/paper/1510.04389/) (2015)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [Dirty Pixels: Towards End-to-End Image Processing and Perception](https://mlsanity.com/paper/1701.06487/) (2017)
- [Learning Sensor Multiplexing Design through Back-propagation](https://mlsanity.com/paper/1605.07078/) (2016)
- [Physics-based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging](https://mlsanity.com/paper/1808.03571/) (2018)
- [Data-Driven Design for Fourier Ptychographic Microscopy](https://mlsanity.com/paper/1904.04175/) (2019)
- [Convolutional neural networks that teach microscopes how to image](https://mlsanity.com/paper/1709.07223/) (2017)
- [Illumination Pattern Design with Deep Learning for Single-Shot Fourier Ptychographic Microscopy](https://mlsanity.com/paper/1810.03481/) (2018)
- [Learning a Probabilistic Strategy for Computational Imaging Sensor Selection](https://mlsanity.com/paper/2003.10424/) (2020)
- [Microscopy Image Restoration with Deep Wiener-Kolmogorov filters](https://mlsanity.com/paper/1911.10989/) (2019)
- [Physics-enhanced machine learning for virtual fluorescence microscopy](https://mlsanity.com/paper/2004.04306/) (2020)
- [Towards an Intelligent Microscope: adaptively learned illumination for optimal sample classification](https://mlsanity.com/paper/1910.10209/) (2019)
- [Multi-element microscope optimization by a learned sensing network with composite physical layers](https://mlsanity.com/paper/2006.15404/) (2020)

## Built on by (most cited first, 1 of 1)

- [From Hours to Seconds: Towards 100x Faster Quantitative Phase Imaging via Differentiable Microscopy](https://mlsanity.com/paper/2205.11521/) (2022)

## Abstract (arXiv)

The trade-off between throughput and image quality is an inherent challenge in microscopy. To improve throughput, compressive imaging under-samples image signals; the images are then computationally reconstructed by solving a regularized inverse problem. Compared to traditional regularizers, Deep Learning based methods have achieved greater success in compression and image quality. However, the information loss in the acquisition process sets the compression bounds. Further improvement in compression, without compromising the reconstruction quality is thus a challenge. In this work, we propose differentiable compressive fluorescence microscopy (partial μ) which includes a realistic generalizable forward model with learnable-physical parameters (e.g. illumination patterns), and a novel physics-inspired inverse model. The cascaded model is end-to-end differentiable and can learn optimal compressive sampling schemes through training data. With our model, we performed thousands of numerical experiments on various compressive microscope configurations. We show that learned sampling encodes important information about the specimens in the illumination field of the microscope allowing higher compression up to × 1024. We further utilize our framework for Task Aware Compression. The experimental results show superior performance on the cell segmentation task.