# Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

arXiv: https://arxiv.org/abs/2204.09871 · HTML: https://mlsanity.com/paper/2204.09871/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 149 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 30 of 47)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering](https://mlsanity.com/paper/2010.09125/) (2020)
- [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://mlsanity.com/paper/2104.01148/) (2021)
- [IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis](https://mlsanity.com/paper/1807.06358/) (2018)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles](https://mlsanity.com/paper/2001.11086/) (2020)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling](https://mlsanity.com/paper/2104.08418/) (2021)
- [Dirty Pixels: Towards End-to-End Image Processing and Perception](https://mlsanity.com/paper/1701.06487/) (2017)
- [Reblur2Deblur: Deblurring Videos via Self-Supervised Learning](https://mlsanity.com/paper/1801.05117/) (2018)
- [NeROIC: Neural Rendering of Objects from Online Image Collections](https://mlsanity.com/paper/2201.02533/) (2022)
- [Geo-Supervised Visual Depth Prediction](https://mlsanity.com/paper/1807.11130/) (2018)
- [Learned reconstructions for practical mask-based lensless imaging](https://mlsanity.com/paper/1908.11502/) (2019)
- [Learning Sensor Multiplexing Design through Back-propagation](https://mlsanity.com/paper/1605.07078/) (2016)
- [Learning Depth from Single Images with Deep Neural Network Embedding Focal Length](https://mlsanity.com/paper/1803.10039/) (2018)
- [Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array](https://mlsanity.com/paper/2006.08565/) (2020)

## Built on by (most cited first, 6 of 6)

- [Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition](https://mlsanity.com/paper/2303.04291/) (2023)
- [Spatially Varying Exposure with 2-by-2 Multiplexing: Optimality and Universality](https://mlsanity.com/paper/2306.17367/) (2023)
- [DISeR: Designing Imaging Systems with Reinforcement Learning](https://mlsanity.com/paper/2309.13851/) (2023)
- [Sensitivity analysis of AI-based algorithms for autonomous driving on optical wavefront aberrations induced by the windshield](https://mlsanity.com/paper/2308.11711/) (2023)
- [γ-Quant: Towards Learnable Quantization for Low-bit Pattern Recognition](https://mlsanity.com/paper/2509.22448/) (2025)
- [RAWDet-7: A Multi-Scenario Benchmark for Object Detection and Description on Quantized RAW Images](https://mlsanity.com/paper/2602.03760/) (2026)

## Abstract (arXiv)

Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-specific camera design. Finally, we share current barriers to progress in end-to-end design, and hypothesize how these barriers can be overcome.