# Spatial-Temporal Transformer for Video Snapshot Compressive Imaging

arXiv: https://arxiv.org/abs/2209.01578 · HTML: https://mlsanity.com/paper/2209.01578/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 72 references that are in the map.
- First posted on arXiv 4 Sep 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)

## Built on by (most cited first, 21 of 21)

- [EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2305.10006/) (2023)
- [HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention](https://mlsanity.com/paper/2402.14185/) (2024)
- [Deep Optics for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2404.05274/) (2024)
- [MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2410.14214/) (2024)
- [Hierarchical Separable Video Transformer for Snapshot Compressive Imaging](https://mlsanity.com/paper/2407.11946/) (2024)
- [SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image](https://mlsanity.com/paper/2403.20018/) (2024)
- [RED-PSM: Regularization by Denoising of Factorized Low Rank Models for Dynamic Imaging](https://mlsanity.com/paper/2304.03483/) (2023)
- [SnapCap: Efficient Snapshot Compressive Video Captioning](https://mlsanity.com/paper/2401.04903/) (2024)
- [Event-Enhanced Snapshot Compressive Videography at 10K FPS](https://mlsanity.com/paper/2404.07551/) (2024)
- [A Decade of Deep Learning for Remote Sensing Spatiotemporal Fusion: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2504.00901/) (2025)
- [MSFA-Frequency-Aware Transformer for Hyperspectral Images Demosaicing](https://mlsanity.com/paper/2303.13404/) (2023)
- [Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms](https://mlsanity.com/paper/2406.03694/) (2024)
- [Towards Real-time Video Compressive Sensing on Mobile Devices](https://mlsanity.com/paper/2408.07530/) (2024)
- [SCIGS: 3D Gaussians Splatting from a Snapshot Compressive Image](https://mlsanity.com/paper/2411.12471/) (2024)
- [Vision without Images: End-to-End Computer Vision from Single Compressive Measurements](https://mlsanity.com/paper/2501.15122/) (2025)
- [TransiT: Transient Transformer for Non-line-of-sight Videography](https://mlsanity.com/paper/2503.11328/) (2025)
- [Deformable Convolution Module with Globally Learned Relative Offsets for Fundus Vessel Segmentation](https://mlsanity.com/paper/2507.18354/) (2025)
- [Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensing](https://mlsanity.com/paper/2509.08228/) (2025)
- [3One2: One-step Regression Plus One-step Diffusion for One-hot Modulation in Dual-path Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2512.17578/) (2025)
- [RayFormer: Modeling Inter- and Intra-Ray Similarity for NeRF-Based Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2604.27702/) (2026)
- [GS2CI: Robust Gaussian Splatting For Snapshot Compressive Imaging via Large Vision Model Priors](https://mlsanity.com/paper/2608.13502/) (2026)

## Abstract (arXiv)

Video snapshot compressive imaging (SCI) captures multiple sequential video frames by a single measurement using the idea of computational imaging. The underlying principle is to modulate high-speed frames through different masks and these modulated frames are summed to a single measurement captured by a low-speed 2D sensor (dubbed optical encoder); following this, algorithms are employed to reconstruct the desired high-speed frames (dubbed software decoder) if needed. In this paper, we consider the reconstruction algorithm in video SCI, i.e., recovering a series of video frames from a compressed measurement. Specifically, we propose a Spatial-Temporal transFormer (STFormer) to exploit the correlation in both spatial and temporal domains. STFormer network is composed of a token generation block, a video reconstruction block, and these two blocks are connected by a series of STFormer blocks. Each STFormer block consists of a spatial self-attention branch, a temporal self-attention branch and the outputs of these two branches are integrated by a fusion network. Extensive results on both simulated and real data demonstrate the state-of-the-art performance of STFormer. The code and models are publicly available at https://github.com/ucaswangls/STFormer.git