# SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers

arXiv: https://arxiv.org/abs/2107.02988 · HTML: https://mlsanity.com/paper/2107.02988/

## Key facts

- Cited by 169 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Deep Learning for Hyperspectral Image Classification: An Overview](https://mlsanity.com/paper/1910.12861/) (2019)
- [Graph Convolutional Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2008.02457/) (2020)
- [An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing](https://mlsanity.com/paper/1810.12000/) (2018)
- [Hyperspectral Image Restoration via Total Variation Regularized Low-rank Tensor Decomposition](https://mlsanity.com/paper/1707.02477/) (2017)
- [Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox)](https://mlsanity.com/paper/2003.02822/) (2020)
- [Cascaded Recurrent Neural Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/1902.10858/) (2019)
- [Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing](https://mlsanity.com/paper/2105.10194/) (2021)
- [Joint and Progressive Subspace Analysis (JPSA) with Spatial-Spectral Manifold Alignment for Semi-Supervised Hyperspectral Dimensionality Reduction](https://mlsanity.com/paper/2009.10003/) (2020)
- [Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification](https://mlsanity.com/paper/1912.08847/) (2019)

## Built on by (most cited first, 30 of 169)

- [UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery](https://mlsanity.com/paper/2109.08937/) (2021)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model](https://mlsanity.com/paper/2406.11519/) (2024)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022)
- [MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification](https://mlsanity.com/paper/2501.04944/) (2025)
- [SpectralEarth: Training Hyperspectral Foundation Models at Scale](https://mlsanity.com/paper/2408.08447/) (2024)
- [SpectralMamba: Efficient Mamba for Hyperspectral Image Classification](https://mlsanity.com/paper/2404.08489/) (2024)
- [Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising](https://mlsanity.com/paper/2304.00844/) (2023)
- [Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution](https://mlsanity.com/paper/2109.02079/) (2021)
- [SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models](https://mlsanity.com/paper/2304.05961/) (2023)
- [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024)
- [Unsupervised Hyperspectral Image Super-Resolution via Self-Supervised Modality Decoupling](https://mlsanity.com/paper/2412.04802/) (2024)
- [Spectral-Spatial Mamba for Hyperspectral Image Classification](https://mlsanity.com/paper/2404.18401/) (2024)
- [Building extraction with vision transformer](https://mlsanity.com/paper/2111.15637/) (2021)
- [SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising](https://mlsanity.com/paper/2405.01726/) (2024)
- [How to Learn More? Exploring Kolmogorov-Arnold Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2406.15719/) (2024)
- [A Comprehensive Survey for Hyperspectral Image Classification: The Evolution from Conventional to Transformers and Mamba Models](https://mlsanity.com/paper/2404.14955/) (2024)
- [Swin MAE: Masked Autoencoders for Small Datasets](https://mlsanity.com/paper/2212.13805/) (2022)
- [3DSS-Mamba: 3D-Spectral-Spatial Mamba for Hyperspectral Image Classification](https://mlsanity.com/paper/2405.12487/) (2024)
- [Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection](https://mlsanity.com/paper/2205.13769/) (2022)
- [Hybrid Spectral Denoising Transformer with Guided Attention](https://mlsanity.com/paper/2303.09040/) (2023)
- [Exploring Multi-Timestep Multi-Stage Diffusion Features for Hyperspectral Image Classification](https://mlsanity.com/paper/2306.08964/) (2023)
- [DualMamba: A Lightweight Spectral-Spatial Mamba-Convolution Network for Hyperspectral Image Classification](https://mlsanity.com/paper/2406.07050/) (2024)
- [HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classification](https://mlsanity.com/paper/2404.00272/) (2024)
- [OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning](https://mlsanity.com/paper/2507.13364/) (2025)
- [3D Instance Segmentation of MVS Buildings](https://mlsanity.com/paper/2112.09902/) (2021)
- [MethaneMapper: Spectral Absorption aware Hyperspectral Transformer for Methane Detection](https://mlsanity.com/paper/2304.02767/) (2023)
- [DCN-T: Dual Context Network with Transformer for Hyperspectral Image Classification](https://mlsanity.com/paper/2304.09915/) (2023)
- [DMSSN: Distilled Mixed Spectral-Spatial Network for Hyperspectral Salient Object Detection](https://mlsanity.com/paper/2404.00694/) (2024)

## Abstract (arXiv)

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called ulSpectralFormer. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse "soft" residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixel- and patch-wise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEETGRSSpectralFormer for the sake of reproducibility.