# Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensing

arXiv: https://arxiv.org/abs/2509.08228 · HTML: https://mlsanity.com/paper/2509.08228/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 77 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Recurrent Video Restoration Transformer with Guided Deformable Attention](https://mlsanity.com/paper/2206.02146/) (2022)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [Efficient and Explicit Modelling of Image Hierarchies for Image Restoration](https://mlsanity.com/paper/2303.00748/) (2023)
- [Rank Minimization for Snapshot Compressive Imaging](https://mlsanity.com/paper/1807.07837/) (2018)
- [Neural Pruning via Growing Regularization](https://mlsanity.com/paper/2012.09243/) (2020)
- [Space-time Mixing Attention for Video Transformer](https://mlsanity.com/paper/2106.05968/) (2021)
- [Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imaging](https://mlsanity.com/paper/2003.13654/) (2020)
- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [Flow-Guided Transformer for Video Inpainting](https://mlsanity.com/paper/2208.06768/) (2022)
- [Coded aperture compressive temporal imaging](https://mlsanity.com/paper/1302.2575/) (2013)
- [MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering](https://mlsanity.com/paper/2212.09522/) (2022)
- [Parameter-Efficient and Student-Friendly Knowledge Distillation](https://mlsanity.com/paper/2205.15308/) (2022)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Plug-and-Play Algorithms for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2101.04822/) (2021)
- [What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective](https://mlsanity.com/paper/2012.02909/) (2020)
- [Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging](https://mlsanity.com/paper/2109.06548/) (2021)
- [Memory-Efficient Network for Large-scale Video Compressive Sensing](https://mlsanity.com/paper/2103.03089/) (2021)
- [MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive Sensing](https://mlsanity.com/paper/2103.01786/) (2021)
- [MAR: Masked Autoencoders for Efficient Action Recognition](https://mlsanity.com/paper/2207.11660/) (2022)

## Abstract (arXiv)

Digital cameras consume ~0.1 microjoule per pixel to capture and encode video, resulting in a power usage of ~20W for a 4K sensor operating at 30 fps. Imagining gigapixel cameras operating at 100-1000 fps, the current processing model is unsustainable. To address this, physical layer compressive measurement has been proposed to reduce power consumption per pixel by 10-100X. Video Snapshot Compressive Imaging (SCI) introduces high frequency modulation in the optical sensor layer to increase effective frame rate. A commonly used sampling strategy of video SCI is Random Sampling (RS) where each mask element value is randomly set to be 0 or 1. Similarly, image inpainting (I2P) has demonstrated that images can be recovered from a fraction of the image pixels. Inspired by I2P, we propose Ultra-Sparse Sampling (USS) regime, where at each spatial location, only one sub-frame is set to 1 and all others are set to 0. We then build a Digital Micro-mirror Device (DMD) encoding system to verify the effectiveness of our USS strategy. Ideally, we can decompose the USS measurement into sub-measurements for which we can utilize I2P algorithms to recover high-speed frames. However, due to the mismatch between the DMD and CCD, the USS measurement cannot be perfectly decomposed. To this end, we propose BSTFormer, a sparse TransFormer that utilizes local Block attention, global Sparse attention, and global Temporal attention to exploit the sparsity of the USS measurement. Extensive results on both simulated and real-world data show that our method significantly outperforms all previous state-of-the-art algorithms. Additionally, an essential advantage of the USS strategy is its higher dynamic range than that of the RS strategy. Finally, from the application perspective, the USS strategy is a good choice to implement a complete video SCI system on chip due to its fixed exposure time.