# Deep Optics for Video Snapshot Compressive Imaging

arXiv: https://arxiv.org/abs/2404.05274 · HTML: https://mlsanity.com/paper/2404.05274/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Rank Minimization for Snapshot Compressive Imaging](https://mlsanity.com/paper/1807.07837/) (2018)
- [Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imaging](https://mlsanity.com/paper/2003.13654/) (2020)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [Plug-and-Play Algorithms for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2101.04822/) (2021)
- [Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging](https://mlsanity.com/paper/2109.06548/) (2021)
- [Memory-Efficient Network for Large-scale Video Compressive Sensing](https://mlsanity.com/paper/2103.03089/) (2021)
- [MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive Sensing](https://mlsanity.com/paper/2103.01786/) (2021)
- [Spatial-Temporal Transformer for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2209.01578/) (2022)
- [DeepBinaryMask: Learning a Binary Mask for Video Compressive Sensing](https://mlsanity.com/paper/1607.03343/) (2016)
- [Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems](https://mlsanity.com/paper/2104.02820/) (2021)

## Built on by (most cited first, 8 of 8)

- [Proximal Algorithm Unrolling: Flexible and Efficient Reconstruction Networks for Single-Pixel Imaging](https://mlsanity.com/paper/2505.23180/) (2025)
- [Hierarchical Separable Video Transformer for Snapshot Compressive Imaging](https://mlsanity.com/paper/2407.11946/) (2024)
- [Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms](https://mlsanity.com/paper/2406.03694/) (2024)
- [Coded-E2LF: Coded Aperture Light Field Imaging from Events](https://mlsanity.com/paper/2602.22620/) (2026)
- [Ultra-Low Complexity On-Orbit Compression for Remote Sensing Imagery via Block Modulated Imaging](https://mlsanity.com/paper/2412.18417/) (2024)
- [Vision without Images: End-to-End Computer Vision from Single Compressive Measurements](https://mlsanity.com/paper/2501.15122/) (2025)
- [3One2: One-step Regression Plus One-step Diffusion for One-hot Modulation in Dual-path Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2512.17578/) (2025)
- [Deep Probabilistic Unfolding for Quantized Compressive Sensing](https://mlsanity.com/paper/2605.11475/) (2026)

## Abstract (arXiv)

Video snapshot compressive imaging (SCI) aims to capture a sequence of video frames with only a single shot of a 2D detector, whose backbones rest in optical modulation patterns (also known as masks) and a computational reconstruction algorithm. Advanced deep learning algorithms and mature hardware are putting video SCI into practical applications. Yet, there are two clouds in the sunshine of SCI: i) low dynamic range as a victim of high temporal multiplexing, and ii) existing deep learning algorithms' degradation on real system. To address these challenges, this paper presents a deep optics framework to jointly optimize masks and a reconstruction network. Specifically, we first propose a new type of structural mask to realize motion-aware and full-dynamic-range measurement. Considering the motion awareness property in measurement domain, we develop an efficient network for video SCI reconstruction using Transformer to capture long-term temporal dependencies, dubbed Res2former. Moreover, sensor response is introduced into the forward model of video SCI to guarantee end-to-end model training close to real system. Finally, we implement the learned structural masks on a digital micro-mirror device. Experimental results on synthetic and real data validate the effectiveness of the proposed framework. We believe this is a milestone for real-world video SCI. The source code and data are available at https://github.com/pwangcs/DeepOpticsSCI.