# MultiScale Spectral-Spatial Convolutional Transformer for Hyperspectral Image Classification

arXiv: https://arxiv.org/abs/2310.18550 · HTML: https://mlsanity.com/paper/2310.18550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [Graph Convolutional Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2008.02457/) (2020)
- [HybridSN: Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification](https://mlsanity.com/paper/1902.06701/) (2019)
- [Three dimensional Deep Learning approach for remote sensing image classification](https://mlsanity.com/paper/1806.05824/) (2018)
- [Cascaded Recurrent Neural Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/1902.10858/) (2019)
- [Multi-scale Dynamic Graph Convolutional Network for Hyperspectral Image Classification](https://mlsanity.com/paper/1905.06133/) (2019)
- [Advances in Hyperspectral Image Classification: Earth monitoring with statistical learning methods](https://mlsanity.com/paper/1310.5107/) (2013)
- [Statistical Loss and Analysis for Deep Learning in Hyperspectral Image Classification](https://mlsanity.com/paper/1912.12385/) (2019)
- [Deep Manifold Embedding for Hyperspectral Image Classification](https://mlsanity.com/paper/1912.11264/) (2019)

## Abstract (arXiv)

Due to the powerful ability in capturing the global information, Transformer has become an alternative architecture of CNNs for hyperspectral image classification. However, general Transformer mainly considers the global spectral information while ignores the multiscale spatial information of the hyperspectral image. In this paper, we propose a multiscale spectral-spatial convolutional Transformer (MultiscaleFormer) for hyperspectral image classification. First, the developed method utilizes multiscale spatial patches as tokens to formulate the spatial Transformer and generates multiscale spatial representation of each band in each pixel. Second, the spatial representation of all the bands in a given pixel are utilized as tokens to formulate the spectral Transformer and generate the multiscale spectral-spatial representation of each pixel. Besides, a modified spectral-spatial CAF module is constructed in the MultiFormer to fuse cross-layer spectral and spatial information. Therefore, the proposed MultiFormer can capture the multiscale spectral-spatial information and provide better performance than most of other architectures for hyperspectral image classification. Experiments are conducted over commonly used real-world datasets and the comparison results show the superiority of the proposed method.