# MMFormer: Multimodal Transformer Using Multiscale Self-Attention for Remote Sensing Image Classification

arXiv: https://arxiv.org/abs/2303.13101 · HTML: https://mlsanity.com/paper/2303.13101/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Three dimensional Deep Learning approach for remote sensing image classification](https://mlsanity.com/paper/1806.05824/) (2018)
- [Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects](https://mlsanity.com/paper/2101.06116/) (2021)
- [ORSIm Detector: A Novel Object Detection Framework in Optical Remote Sensing Imagery Using Spatial-Frequency Channel Features](https://mlsanity.com/paper/1901.07925/) (2019)
- [HetConv: Heterogeneous Kernel-Based Convolutions for Deep CNNs](https://mlsanity.com/paper/1903.04120/) (2019)
- [Fourier-based Rotation-invariant Feature Boosting: An Efficient Framework for Geospatial Object Detection](https://mlsanity.com/paper/1905.11074/) (2019)
- [Spectral Superresolution of Multispectral Imagery with Joint Sparse and Low-Rank Learning](https://mlsanity.com/paper/2007.14006/) (2020)

## Abstract (arXiv)

To benefit the complementary information between heterogeneous data, we introduce a new Multimodal Transformer (MMFormer) for Remote Sensing (RS) image classification using Hyperspectral Image (HSI) accompanied by another source of data such as Light Detection and Ranging (LiDAR). Compared with traditional Vision Transformer (ViT) lacking inductive biases of convolutions, we first introduce convolutional layers to our MMFormer to tokenize patches from multimodal data of HSI and LiDAR. Then we propose a Multi-scale Multi-head Self-Attention (MSMHSA) module to address the problem of compatibility which often limits to fuse HSI with high spectral resolution and LiDAR with relatively low spatial resolution. The proposed MSMHSA module can incorporate HSI to LiDAR data in a coarse-to-fine manner enabling us to learn a fine-grained representation. Extensive experiments on widely used benchmarks (e.g., Trento and MUUFL) demonstrate the effectiveness and superiority of our proposed MMFormer for RS image classification.