# Learning Sensor Multiplexing Design through Back-propagation

arXiv: https://arxiv.org/abs/1605.07078 · HTML: https://mlsanity.com/paper/1605.07078/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2016.

## Built on by (most cited first, 28 of 28)

- [Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems](https://mlsanity.com/paper/2104.02820/) (2021)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Learning Spatially Varying Pixel Exposures for Motion Deblurring](https://mlsanity.com/paper/2204.07267/) (2022)
- [CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo](https://mlsanity.com/paper/2104.04641/) (2021)
- [Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile Vision](https://mlsanity.com/paper/2109.08880/) (2021)
- [Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging](https://mlsanity.com/paper/2204.09871/) (2022)
- [Differentiable Sensor Layouts for End-to-End Learning of Task-Specific Camera Parameters](https://mlsanity.com/paper/2304.14736/) (2023)
- [Spatially Varying Exposure with 2-by-2 Multiplexing: Optimality and Universality](https://mlsanity.com/paper/2306.17367/) (2023)
- [DISeR: Designing Imaging Systems with Reinforcement Learning](https://mlsanity.com/paper/2309.13851/) (2023)
- [PUERT: Probabilistic Under-sampling and Explicable Reconstruction Network for CS-MRI](https://mlsanity.com/paper/2204.11189/) (2022)
- [Deep Optical Coding Design in Computational Imaging](https://mlsanity.com/paper/2207.00164/) (2022)
- [Acquiring a Dynamic Light Field through a Single-Shot Coded Image](https://mlsanity.com/paper/2204.12089/) (2022)
- [Optimizing Camera Configurations for Multi-View Pedestrian Detection](https://mlsanity.com/paper/2312.02144/) (2023)
- [SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera](https://mlsanity.com/paper/2406.04287/) (2024)
- [Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation](https://mlsanity.com/paper/2603.26528/) (2026)
- [SaccadeCam: Adaptive Visual Attention for Monocular Depth Sensing](https://mlsanity.com/paper/2103.12981/) (2021)
- [Joint Optimization of Hadamard Sensing and Reconstruction in Compressed Sensing Fluorescence Microscopy](https://mlsanity.com/paper/2105.07961/) (2021)
- [Differentiable Microscopy Designs an All Optical Phase Retrieval Microscope](https://mlsanity.com/paper/2203.14944/) (2022)
- [Differentiable Microscopy for Content and Task Aware Compressive Fluorescence Imaging](https://mlsanity.com/paper/2203.14945/) (2022)
- [Latent Space Imaging](https://mlsanity.com/paper/2407.07052/) (2024)
- [Optimal OnTheFly Feedback Control of Event Sensors](https://mlsanity.com/paper/2408.12976/) (2024)
- [Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging](https://mlsanity.com/paper/2507.22523/) (2025)
- [Automatic calibration of time of flight based non-line-of-sight reconstruction](https://mlsanity.com/paper/2105.10603/) (2021)
- [Noise-Adaptive Intelligent Programmable Meta-Imager](https://mlsanity.com/paper/2208.10171/) (2022)
- [LIPIDS: Learning-based Illumination Planning In Discretized (Light) Space for Photometric Stereo](https://mlsanity.com/paper/2409.02716/) (2024)
- [Learning to Sense for Driving: Joint Optics-Sensor-Model Co-Design for Semantic Segmentation](https://mlsanity.com/paper/2512.20815/) (2025)
- [Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation](https://mlsanity.com/paper/2606.24096/) (2026)
- [Evaluator-Dependent Patient-Adaptive ECG Lead-Channel Allocation](https://mlsanity.com/paper/2608.26827/) (2026)

## Abstract (arXiv)

Recent progress on many imaging and vision tasks has been driven by the use of deep feed-forward neural networks, which are trained by propagating gradients of a loss defined on the final output, back through the network up to the first layer that operates directly on the image. We propose back-propagating one step further---to learn camera sensor designs jointly with networks that carry out inference on the images they capture. In this paper, we specifically consider the design and inference problems in a typical color camera---where the sensor is able to measure only one color channel at each pixel location, and computational inference is required to reconstruct a full color image. We learn the camera sensor's color multiplexing pattern by encoding it as layer whose learnable weights determine which color channel, from among a fixed set, will be measured at each location. These weights are jointly trained with those of a reconstruction network that operates on the corresponding sensor measurements to produce a full color image. Our network achieves significant improvements in accuracy over the traditional Bayer pattern used in most color cameras. It automatically learns to employ a sparse color measurement approach similar to that of a recent design, and moreover, improves upon that design by learning an optimal layout for these measurements.