# Multimodal Hyperspectral Image Classification via Interconnected Fusion

arXiv: https://arxiv.org/abs/2304.00495 · HTML: https://mlsanity.com/paper/2304.00495/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 2 Apr 2023.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review](https://mlsanity.com/paper/2205.01380/) (2022)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Going Deeper with Contextual CNN for Hyperspectral Image Classification](https://mlsanity.com/paper/1604.03519/) (2016)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [ConvNets vs. Transformers: Whose Visual Representations are More Transferable?](https://mlsanity.com/paper/2108.05305/) (2021)
- [Classification of Hyperspectral and LiDAR Data Using Coupled CNNs](https://mlsanity.com/paper/2002.01144/) (2020)

## Built on by (most cited first, 1 of 1)

- [AI-Driven HSI: Multimodality, Fusion, Challenges, and the Deep Learning Revolution](https://mlsanity.com/paper/2502.06894/) (2025)

## Abstract (arXiv)

Existing multiple modality fusion methods, such as concatenation, summation, and encoder-decoder-based fusion, have recently been employed to combine modality characteristics of Hyperspectral Image (HSI) and Light Detection And Ranging (LiDAR). However, these methods consider the relationship of HSI-LiDAR signals from limited perspectives. More specifically, they overlook the contextual information across modalities of HSI and LiDAR and the intra-modality characteristics of LiDAR. In this paper, we provide a new insight into feature fusion to explore the relationships across HSI and LiDAR modalities comprehensively. An Interconnected Fusion (IF) framework is proposed. Firstly, the center patch of the HSI input is extracted and replicated to the size of the HSI input. Then, nine different perspectives in the fusion matrix are generated by calculating self-attention and cross-attention among the replicated center patch, HSI input, and corresponding LiDAR input. In this way, the intra- and inter-modality characteristics can be fully exploited, and contextual information is considered in both intra-modality and inter-modality manner. These nine interrelated elements in the fusion matrix can complement each other and eliminate biases, which can generate a multi-modality representation for classification accurately. Extensive experiments have been conducted on three widely used datasets: Trento, MUUFL, and Houston. The IF framework achieves state-of-the-art results on these datasets compared to existing approaches.