# Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

arXiv: https://arxiv.org/abs/2203.04845 · HTML: https://mlsanity.com/paper/2203.04845/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 107 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)

## Built on by (most cited first, 30 of 67)

- [MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction](https://mlsanity.com/paper/2204.07908/) (2022)
- [Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging](https://mlsanity.com/paper/2205.10102/) (2022)
- [Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks](https://mlsanity.com/paper/2503.16930/) (2025)
- [Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising](https://mlsanity.com/paper/2304.00844/) (2023)
- [Binarized Spectral Compressive Imaging](https://mlsanity.com/paper/2305.10299/) (2023)
- [Residual Degradation Learning Unfolding Framework with Mixing Priors across Spectral and Spatial for Compressive Spectral Imaging](https://mlsanity.com/paper/2211.06891/) (2022)
- [Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance](https://mlsanity.com/paper/2505.02109/) (2025)
- [Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2312.01061/) (2023)
- [Hybrid Spectral Denoising Transformer with Guided Attention](https://mlsanity.com/paper/2303.09040/) (2023)
- [Invertible Diffusion Models for Compressed Sensing](https://mlsanity.com/paper/2403.17006/) (2024)
- [Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration](https://mlsanity.com/paper/2205.10195/) (2022)
- [Hyperspectral Image Reconstruction via Combinatorial Embedding of Cross-Channel Spatio-Spectral Clues](https://mlsanity.com/paper/2312.11119/) (2023)
- [In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging](https://mlsanity.com/paper/2312.13319/) (2023)
- [freePruner: A Training-free Approach for Large Multimodal Model Acceleration](https://mlsanity.com/paper/2411.15446/) (2024)
- [Aperture Diffraction for Compact Snapshot Spectral Imaging](https://mlsanity.com/paper/2309.16372/) (2023)
- [Progressive Content-aware Coded Hyperspectral Compressive Imaging](https://mlsanity.com/paper/2303.09773/) (2023)
- [DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model](https://mlsanity.com/paper/2311.11417/) (2023)
- [Hyperspectral Image Denoising via Spatial-Spectral Recurrent Transformer](https://mlsanity.com/paper/2401.03885/) (2023)
- [Dispersed Structured Light for Hyperspectral 3D Imaging](https://mlsanity.com/paper/2311.18287/) (2023)
- [GMSR:Gradient-Guided Mamba for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2405.07777/) (2024)
- [Prototypical Transformer as Unified Motion Learners](https://mlsanity.com/paper/2406.01559/) (2024)
- [Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging](https://mlsanity.com/paper/2306.01176/) (2023)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [A Simple Adaptive Unfolding Network for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2301.10208/) (2023)
- [SST-ReversibleNet: Reversible-prior-based Spectral-Spatial Transformer for Efficient Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2305.04054/) (2023)
- [Dual Hyperspectral Mamba for Efficient Spectral Compressive Imaging](https://mlsanity.com/paper/2406.00449/) (2024)
- [Exploiting Frequency Correlation for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2406.00683/) (2024)
- [Window-based Channel Attention for Wavelet-enhanced Learned Image Compression](https://mlsanity.com/paper/2409.14090/) (2024)
- [Mixed-granularity Implicit Representation for Continuous Hyperspectral Compressive Reconstruction](https://mlsanity.com/paper/2503.12783/) (2025)
- [TreEnhance: A Tree Search Method For Low-Light Image Enhancement](https://mlsanity.com/paper/2205.12639/) (2022)

## Abstract (arXiv)

Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) between some tokens that are unrelated in content. This does not fit the spatially sparse nature of HSI signals and limits the model scalability. In this paper, we propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms state-of-the-art methods while requiring cheaper computational costs. The code and models will be released at https://github.com/caiyuanhao1998/MST