# The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection

arXiv: https://arxiv.org/abs/2212.04441 · HTML: https://mlsanity.com/paper/2212.04441/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 46 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)

## Built on by (most cited first, 15 of 15)

- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers](https://mlsanity.com/paper/2306.12992/) (2023)
- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [Task-Driven Lens Design](https://mlsanity.com/paper/2305.17185/) (2023)
- [Exploring Quasi-Global Solutions to Compound Lens Based Computational Imaging Systems](https://mlsanity.com/paper/2404.19201/) (2024)
- [A Differentiable Wave Optics Model for End-to-End Computational Imaging System Optimization](https://mlsanity.com/paper/2412.09774/) (2024)
- [Minimalist Vision with Freeform Pixels](https://mlsanity.com/paper/2501.00142/) (2024)
- [Successive optimization of optics and post-processing with differentiable coherent PSF operator and field information](https://mlsanity.com/paper/2412.14603/) (2024)
- [TaCOS: Task-Specific Camera Optimization with Simulation](https://mlsanity.com/paper/2404.11031/) (2024)
- [Simulating Dual-Pixel Images From Ray Tracing For Depth Estimation](https://mlsanity.com/paper/2503.11213/) (2025)
- [Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camera Design](https://mlsanity.com/paper/2412.19439/) (2024)
- [Automated design of compound lenses with discrete-continuous optimization](https://mlsanity.com/paper/2509.23572/) (2025)
- [JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithms](https://mlsanity.com/paper/2512.06763/) (2025)
- [Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation](https://mlsanity.com/paper/2606.24096/) (2026)
- [LensDesigner: A Self-Improving Agent for Optical Lens Design](https://mlsanity.com/paper/2609.30450/) (2026)

## Abstract (arXiv)

Most camera lens systems are designed in isolation, separately from downstream computer vision methods. Recently, joint optimization approaches that design lenses alongside other components of the image acquisition and processing pipeline -- notably, downstream neural networks -- have achieved improved imaging quality or better performance on vision tasks. However, these existing methods optimize only a subset of lens parameters and cannot optimize glass materials given their categorical nature. In this work, we develop a differentiable spherical lens simulation model that accurately captures geometrical aberrations. We propose an optimization strategy to address the challenges of lens design -- notorious for non-convex loss function landscapes and many manufacturing constraints -- that are exacerbated in joint optimization tasks. Specifically, we introduce quantized continuous glass variables to facilitate the optimization and selection of glass materials in an end-to-end design context, and couple this with carefully designed constraints to support manufacturability. In automotive object detection, we report improved detection performance over existing designs even when simplifying designs to two- or three-element lenses, despite significantly degrading the image quality.