# MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction

arXiv: https://arxiv.org/abs/2204.07908 · HTML: https://mlsanity.com/paper/2204.07908/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 90 references that are in the map.
- First posted on arXiv 17 Apr 2022.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Learning Enriched Features for Real Image Restoration and Enhancement](https://mlsanity.com/paper/2003.06792/) (2020)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [HINet: Half Instance Normalization Network for Image Restoration](https://mlsanity.com/paper/2105.06086/) (2021)
- [Unfolding the Alternating Optimization for Blind Super Resolution](https://mlsanity.com/paper/2010.02631/) (2020)
- [TransPose: Keypoint Localization via Transformer](https://mlsanity.com/paper/2012.14214/) (2020)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2111.07910/) (2021)
- [TokenPose: Learning Keypoint Tokens for Human Pose Estimation](https://mlsanity.com/paper/2104.03516/) (2021)
- [Rank Minimization for Snapshot Compressive Imaging](https://mlsanity.com/paper/1807.07837/) (2018)

## Built on by (most cited first, 30 of 72)

- [Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging](https://mlsanity.com/paper/2205.10102/) (2022)
- [NTIRE 2024 Challenge on Low Light Image Enhancement: Methods and Results](https://mlsanity.com/paper/2404.14248/) (2024)
- [Binarized Spectral Compressive Imaging](https://mlsanity.com/paper/2305.10299/) (2023)
- [Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration](https://mlsanity.com/paper/2205.10195/) (2022)
- [Hyperspectral Image Reconstruction via Combinatorial Embedding of Cross-Channel Spatio-Spectral Clues](https://mlsanity.com/paper/2312.11119/) (2023)
- [Multiple Instance Learning with Mixed Supervision in Gleason Grading](https://mlsanity.com/paper/2206.12798/) (2022)
- [In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging](https://mlsanity.com/paper/2312.13319/) (2023)
- [Aperture Diffraction for Compact Snapshot Spectral Imaging](https://mlsanity.com/paper/2309.16372/) (2023)
- [PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation](https://mlsanity.com/paper/2411.15922/) (2024)
- [Progressive Content-aware Coded Hyperspectral Compressive Imaging](https://mlsanity.com/paper/2303.09773/) (2023)
- [Mobile Image Restoration via Prior Quantization](https://mlsanity.com/paper/2305.05899/) (2023)
- [DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model](https://mlsanity.com/paper/2311.11417/) (2023)
- [Hyperspectral Image Super-Resolution via Dual-domain Network Based on Hybrid Convolution](https://mlsanity.com/paper/2304.04589/) (2023)
- [Limitations of Data-Driven Spectral Reconstruction -- An Optics-Aware Analysis](https://mlsanity.com/paper/2401.03835/) (2024)
- [Pansharpening for Thin-Cloud Contaminated Remote Sensing Images: A Unified Framework and Benchmark Dataset](https://mlsanity.com/paper/2603.14952/) (2026)
- [GMSR:Gradient-Guided Mamba for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2405.07777/) (2024)
- [Color Matching Using Hypernetwork-Based Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2503.11781/) (2025)
- [HADAR-Based Thermal Infrared Hyperspectral Image Restoration](https://mlsanity.com/paper/2605.13664/) (2026)
- [Learning to Recover Spectral Reflectance from RGB Images](https://mlsanity.com/paper/2304.02162/) (2023)
- [SST-ReversibleNet: Reversible-prior-based Spectral-Spatial Transformer for Efficient Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2305.04054/) (2023)
- [Dual Hyperspectral Mamba for Efficient Spectral Compressive Imaging](https://mlsanity.com/paper/2406.00449/) (2024)
- [RGB to Hyperspectral: Spectral Reconstruction for Enhanced Surgical Imaging](https://mlsanity.com/paper/2410.13570/) (2024)
- [One-shot neural band selection for spectral recovery](https://mlsanity.com/paper/2305.09236/) (2023)
- [MLP-AMDC: An MLP Architecture for Adaptive-Mask-based Dual-Camera snapshot hyperspectral imaging](https://mlsanity.com/paper/2310.08002/) (2023)
- [Comparative Analysis of Hyperspectral Image Reconstruction Using Deep Learning for Agricultural and Biological Applications](https://mlsanity.com/paper/2405.13331/) (2024)
- [Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusion](https://mlsanity.com/paper/2412.02798/) (2024)
- [SpecDM: Hyperspectral Dataset Synthesis with Pixel-level Semantic Annotations](https://mlsanity.com/paper/2502.17056/) (2025)
- [Multispectral Demosaicing via Dual Cameras](https://mlsanity.com/paper/2503.22026/) (2025)
- [FRN: Fractal-Based Recursive Spectral Reconstruction Network](https://mlsanity.com/paper/2505.15439/) (2025)
- [PCMamba: Physics-Informed Cross-Modal State Space Model for Dual-Camera Compressive Hyperspectral Imaging](https://mlsanity.com/paper/2505.16373/) (2025)

## Abstract (arXiv)

Existing leading methods for spectral reconstruction (SR) focus on designing deeper or wider convolutional neural networks (CNNs) to learn the end-to-end mapping from the RGB image to its hyperspectral image (HSI). These CNN-based methods achieve impressive restoration performance while showing limitations in capturing the long-range dependencies and self-similarity prior. To cope with this problem, we propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++), for efficient spectral reconstruction. In particular, we employ Spectral-wise Multi-head Self-attention (S-MSA) that is based on the HSI spatially sparse while spectrally self-similar nature to compose the basic unit, Spectral-wise Attention Block (SAB). Then SABs build up Single-stage Spectral-wise Transformer (SST) that exploits a U-shaped structure to extract multi-resolution contextual information. Finally, our MST++, cascaded by several SSTs, progressively improves the reconstruction quality from coarse to fine. Comprehensive experiments show that our MST++ significantly outperforms other state-of-the-art methods. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place. Code and pre-trained models are publicly available at https://github.com/caiyuanhao1998/MST-plus-plus.