# More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

arXiv: https://arxiv.org/abs/2008.05457 · HTML: https://mlsanity.com/paper/2008.05457/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2020.

## Built on by (most cited first, 30 of 126)

- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [UIU-Net: U-Net in U-Net for Infrared Small Object Detection](https://mlsanity.com/paper/2212.00968/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval](https://mlsanity.com/paper/2105.07921/) (2021)
- [Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review](https://mlsanity.com/paper/2205.01380/) (2022)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks](https://mlsanity.com/paper/2309.16499/) (2023)
- [Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects](https://mlsanity.com/paper/2101.06116/) (2021)
- [SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery](https://mlsanity.com/paper/2209.13351/) (2022)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [Generate Your Own Scotland: Satellite Image Generation Conditioned on Maps](https://mlsanity.com/paper/2308.16648/) (2023)
- [Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing](https://mlsanity.com/paper/2105.10194/) (2021)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2206.13114/) (2022)
- [FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients](https://mlsanity.com/paper/2401.02433/) (2023)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [Remote Sensing Object Detection Meets Deep Learning: A Meta-review of Challenges and Advances](https://mlsanity.com/paper/2309.06751/) (2023)
- [Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning](https://mlsanity.com/paper/2312.01191/) (2023)
- [SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detection](https://mlsanity.com/paper/2403.05416/) (2024)
- [Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images](https://mlsanity.com/paper/2112.01932/) (2021)
- [Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection](https://mlsanity.com/paper/2205.13769/) (2022)
- [FPS-Net: A Convolutional Fusion Network for Large-Scale LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2103.00738/) (2021)
- [3D Instance Segmentation of MVS Buildings](https://mlsanity.com/paper/2112.09902/) (2021)
- [Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification](https://mlsanity.com/paper/2301.03335/) (2023)
- [Convolutional Neural Networks Rarely Learn Shape for Semantic Segmentation](https://mlsanity.com/paper/2305.06568/) (2023)
- [Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation](https://mlsanity.com/paper/2404.04531/) (2024)
- [Segment Anything with Multiple Modalities](https://mlsanity.com/paper/2408.09085/) (2024)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)

## Abstract (arXiv)

Classification and identification of the materials lying over or beneath the Earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS) and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML) -- cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on "what", "where", and "how" to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS datasets. Furthermore, the codes and datasets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.