# Multimodal Fusion Transformer for Remote Sensing Image Classification

arXiv: https://arxiv.org/abs/2203.16952 · HTML: https://mlsanity.com/paper/2203.16952/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 77 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Graph Convolutional Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2008.02457/) (2020)
- [HybridSN: Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification](https://mlsanity.com/paper/1902.06701/) (2019)
- [An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing](https://mlsanity.com/paper/1810.12000/) (2018)
- [Three dimensional Deep Learning approach for remote sensing image classification](https://mlsanity.com/paper/1806.05824/) (2018)
- [Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects](https://mlsanity.com/paper/2101.06116/) (2021)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox)](https://mlsanity.com/paper/2003.02822/) (2020)
- [diffGrad: An Optimization Method for Convolutional Neural Networks](https://mlsanity.com/paper/1909.11015/) (2019)
- [Classification of Hyperspectral and LiDAR Data Using Coupled CNNs](https://mlsanity.com/paper/2002.01144/) (2020)
- [ORSIm Detector: A Novel Object Detection Framework in Optical Remote Sensing Imagery Using Spatial-Frequency Channel Features](https://mlsanity.com/paper/1901.07925/) (2019)
- [HetConv: Heterogeneous Kernel-Based Convolutions for Deep CNNs](https://mlsanity.com/paper/1903.04120/) (2019)
- [Joint and Progressive Subspace Analysis (JPSA) with Spatial-Spectral Manifold Alignment for Semi-Supervised Hyperspectral Dimensionality Reduction](https://mlsanity.com/paper/2009.10003/) (2020)
- [Fourier-based Rotation-invariant Feature Boosting: An Efficient Framework for Geospatial Object Detection](https://mlsanity.com/paper/1905.11074/) (2019)
- [Spectral Superresolution of Multispectral Imagery with Joint Sparse and Low-Rank Learning](https://mlsanity.com/paper/2007.14006/) (2020)

## Built on by (most cited first, 30 of 64)

- [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022)
- [SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints](https://mlsanity.com/paper/2312.02464/) (2023)
- [Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation](https://mlsanity.com/paper/2310.03986/) (2023)
- [Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2206.13114/) (2022)
- [SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification](https://mlsanity.com/paper/2311.04442/) (2023)
- [FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients](https://mlsanity.com/paper/2401.02433/) (2023)
- [Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution with Subpixel Fusion](https://mlsanity.com/paper/2205.03742/) (2022)
- [Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning](https://mlsanity.com/paper/2312.01191/) (2023)
- [Deep Learning Techniques for Hyperspectral Image Analysis in Agriculture: A Review](https://mlsanity.com/paper/2304.13880/) (2023)
- [FedFusion: Manifold Driven Federated Learning for Multi-satellite and Multi-modality Fusion](https://mlsanity.com/paper/2311.09540/) (2023)
- [MVP: Meta Visual Prompt Tuning for Few-Shot Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2309.09276/) (2023)
- [Dual Selective Fusion Transformer Network for Hyperspectral Image Classification](https://mlsanity.com/paper/2410.03171/) (2024)
- [Spatial Gated Multi-Layer Perceptron for Land Use and Land Cover Mapping](https://mlsanity.com/paper/2308.05235/) (2023)
- [Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classification](https://mlsanity.com/paper/2401.03179/) (2024)
- [CrossFuse: Learning Infrared and Visible Image Fusion by Cross-Sensor Top-K Vision Alignment and Beyond](https://mlsanity.com/paper/2502.14493/) (2025)
- [DocBinFormer: A Two-Level Transformer Network for Effective Document Image Binarization](https://mlsanity.com/paper/2312.03568/) (2023)
- [BihoT: A Large-Scale Dataset and Benchmark for Hyperspectral Camouflaged Object Tracking](https://mlsanity.com/paper/2408.12232/) (2024)
- [MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing](https://mlsanity.com/paper/2603.17528/) (2026)
- [Incomplete Multimodal Learning for Remote Sensing Data Fusion](https://mlsanity.com/paper/2304.11381/) (2023)
- [Irregular Tensor Low-Rank Representation for Hyperspectral Image Representation](https://mlsanity.com/paper/2410.18388/) (2024)
- [From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing](https://mlsanity.com/paper/2411.05826/) (2024)
- [M3amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classification](https://mlsanity.com/paper/2503.06446/) (2025)
- [Semantic-Spatial Feature Fusion with Dynamic Graph Refinement for Remote Sensing Image Captioning](https://mlsanity.com/paper/2503.23453/) (2025)
- [Multimodal Hyperspectral Image Classification via Interconnected Fusion](https://mlsanity.com/paper/2304.00495/) (2023)
- [Neighborhood Attention Makes the Encoder of ResUNet Stronger for Accurate Road Extraction](https://mlsanity.com/paper/2306.04947/) (2023)
- [On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery](https://mlsanity.com/paper/2409.12026/) (2024)
- [Multi-modal Data Fusion and Deep Ensemble Learning for Accurate Crop Yield Prediction](https://mlsanity.com/paper/2502.06062/) (2025)
- [Language-Informed Hyperspectral Image Synthesis for Imbalanced-Small Sample Classification via Semi-Supervised Conditional Diffusion Model](https://mlsanity.com/paper/2502.19700/) (2025)
- [A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware](https://mlsanity.com/paper/2505.11066/) (2025)
- [Electrolyzers-HSI: Close-Range Multi-Scene Hyperspectral Imaging Benchmark Dataset](https://mlsanity.com/paper/2505.20507/) (2025)

## Abstract (arXiv)

Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a distinctive representation in a reduced and hierarchical feature space. Extensive experiments are carried out on widely used benchmark datasets i.e., the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at urlhttps://github.com/AnkurDeria/MFT.