# JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithms

arXiv: https://arxiv.org/abs/2512.06763 · HTML: https://mlsanity.com/paper/2512.06763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 7 Dec 2025.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Evolution-Guided Policy Gradient in Reinforcement Learning](https://mlsanity.com/paper/1805.07917/) (2018)
- [Fast Optical Flow using Dense Inverse Search](https://mlsanity.com/paper/1603.03590/) (2016)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [Dirty Pixels: Towards End-to-End Image Processing and Perception](https://mlsanity.com/paper/1701.06487/) (2017)
- [Collaborative Evolutionary Reinforcement Learning](https://mlsanity.com/paper/1905.00976/) (2019)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Learning Depth from Single Images with Deep Neural Network Embedding Focal Length](https://mlsanity.com/paper/1803.10039/) (2018)
- [DynamicISP: Dynamically Controlled Image Signal Processor for Image Recognition](https://mlsanity.com/paper/2211.01146/) (2022)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection](https://mlsanity.com/paper/2212.04441/) (2022)
- [Learned Camera Gain and Exposure Control for Improved Visual Feature Detection and Matching](https://mlsanity.com/paper/2102.04341/) (2021)
- [Learning Spatially Varying Pixel Exposures for Motion Deblurring](https://mlsanity.com/paper/2204.07267/) (2022)
- [Soft Prototyping Camera Designs for Car Detection Based on a Convolutional Neural Network](https://mlsanity.com/paper/1910.10916/) (2019)
- [DISeR: Designing Imaging Systems with Reinforcement Learning](https://mlsanity.com/paper/2309.13851/) (2023)
- [A system for generating complex physically accurate sensor images for automotive applications](https://mlsanity.com/paper/1902.04258/) (2019)
- [All You Need is RAW: Defending Against Adversarial Attacks with Camera Image Pipelines](https://mlsanity.com/paper/2112.09219/) (2021)
- [Optimizing Camera Configurations for Multi-View Pedestrian Detection](https://mlsanity.com/paper/2312.02144/) (2023)
- [Multiplexed Illumination for Classifying Visually Similar Objects](https://mlsanity.com/paper/2009.11084/) (2020)
- [Using simulation to quantify the performance of automotive perception systems](https://mlsanity.com/paper/2303.00983/) (2023)
- [TaCOS: Task-Specific Camera Optimization with Simulation](https://mlsanity.com/paper/2404.11031/) (2024)
- [What if Eye...? Computationally Recreating Vision Evolution](https://mlsanity.com/paper/2501.15001/) (2025)

## Abstract (arXiv)

The quality of captured images strongly influences the performance of downstream perception tasks. Recent works on co-designing camera systems with perception tasks have shown improved task performance. However, most prior approaches focus on optimising fixed camera parameters set at manufacturing, while many parameters, such as exposure settings, require adaptive control at runtime. This paper introduces a method that jointly optimises camera hardware and adaptive camera control algorithms with downstream vision tasks. We present a unified optimisation framework that integrates gradient-based and derivative-free methods, enabling support for both continuous and discrete parameters, non-differentiable image formation processes, and neural network-based adaptive control algorithms. To address non-differentiable effects such as motion blur, we propose DF-Grad, a hybrid optimisation strategy that trains adaptive control networks using signals from a derivative-free optimiser alongside unsupervised task-driven learning. Experiments show that our method outperforms baselines that optimise static and dynamic parameters separately, particularly under challenging conditions such as low light and fast motion. These results demonstrate that jointly optimising hardware parameters and adaptive control algorithms improves perception performance and provides a unified approach to task-driven camera system design.