# Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

arXiv: https://arxiv.org/abs/2205.01380 · HTML: https://mlsanity.com/paper/2205.01380/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 216 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Deep learning in remote sensing: a review](https://mlsanity.com/paper/1710.03959/) (2017)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Graph Convolutional Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2008.02457/) (2020)
- [Hyperspectral pansharpening: a review](https://mlsanity.com/paper/1504.04531/) (2015)
- [A Multi-Scale and Multi-Depth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening](https://mlsanity.com/paper/1712.09809/) (2017)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox)](https://mlsanity.com/paper/2003.02822/) (2020)
- [Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net](https://mlsanity.com/paper/1901.03281/) (2019)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening](https://mlsanity.com/paper/2203.02503/) (2022)
- [Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution](https://mlsanity.com/paper/2109.02079/) (2021)
- [Pansharpening via Detail Injection Based Convolutional Neural Networks](https://mlsanity.com/paper/1806.08898/) (2018)
- [Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution](https://mlsanity.com/paper/2007.05230/) (2020)
- [Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/1804.05042/) (2018)
- [Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution](https://mlsanity.com/paper/1905.01752/) (2019)
- [Remote Sensing Image Fusion Based on Two-stream Fusion Network](https://mlsanity.com/paper/1711.02549/) (2017)
- [Classification of Hyperspectral and LiDAR Data Using Coupled CNNs](https://mlsanity.com/paper/2002.01144/) (2020)
- [Boosting the accuracy of multi-spectral image pan-sharpening by learning a deep residual network](https://mlsanity.com/paper/1705.07556/) (2017)
- [Recent Advances and New Guidelines on Hyperspectral and Multispectral Image Fusion](https://mlsanity.com/paper/2008.03426/) (2020)
- [Target-adaptive CNN-based pansharpening](https://mlsanity.com/paper/1709.06054/) (2017)
- [Guided Deep Decoder: Unsupervised Image Pair Fusion](https://mlsanity.com/paper/2007.11766/) (2020)
- [PanFormer: a Transformer Based Model for Pan-sharpening](https://mlsanity.com/paper/2203.02916/) (2022)
- [Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super-Resolution](https://mlsanity.com/paper/2007.14007/) (2020)
- [Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network](https://mlsanity.com/paper/1706.05249/) (2017)
- [Unsupervised and Unregistered Hyperspectral Image Super-Resolution with Mutual Dirichlet-Net](https://mlsanity.com/paper/1904.12175/) (2019)
- [Hyperspectral Image Super-resolution via Deep Progressive Zero-centric Residual Learning](https://mlsanity.com/paper/2006.10300/) (2020)
- [Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction](https://mlsanity.com/paper/2107.02630/) (2021)
- [Towards seamless multi-view scene analysis from satellite to street-level](https://mlsanity.com/paper/1705.08101/) (2017)
- [Hyperspectral Image Super-Resolution via Deep Prior Regularization with Parameter Estimation](https://mlsanity.com/paper/2009.04237/) (2020)

## Built on by (most cited first, 30 of 73)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [TerraMind: Large-Scale Generative Multimodality for Earth Observation](https://mlsanity.com/paper/2504.11171/) (2025)
- [Mission Critical -- Satellite Data is a Distinct Modality in Machine Learning](https://mlsanity.com/paper/2402.01444/) (2024)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [Generate Your Own Scotland: Satellite Image Generation Conditioned on Maps](https://mlsanity.com/paper/2308.16648/) (2023)
- [Unsupervised Hyperspectral Image Super-Resolution via Self-Supervised Modality Decoupling](https://mlsanity.com/paper/2412.04802/) (2024)
- [Better, Not Just More: Data-Centric Machine Learning for Earth Observation](https://mlsanity.com/paper/2312.05327/) (2023)
- [Multi-modal learning for geospatial vegetation forecasting](https://mlsanity.com/paper/2303.16198/) (2023)
- [A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2410.11160/) (2024)
- [Conditional Controllable Image Fusion](https://mlsanity.com/paper/2411.01573/) (2024)
- [ASANet: Asymmetric Semantic Aligning Network for RGB and SAR image land cover classification](https://mlsanity.com/paper/2412.02044/) (2024)
- [The Potential of Visual ChatGPT For Remote Sensing](https://mlsanity.com/paper/2304.13009/) (2023)
- [Cross-Scope Spatial-Spectral Information Aggregation for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/2311.17340/) (2023)
- [Tensor Decompositions for Hyperspectral Data Processing in Remote Sensing: A Comprehensive Review](https://mlsanity.com/paper/2205.06407/) (2022)
- [A Spectral Diffusion Prior for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/2311.08955/) (2023)
- [Adaptive Fusion of Multi-view Remote Sensing data for Optimal Sub-field Crop Yield Prediction](https://mlsanity.com/paper/2401.11844/) (2024)
- [Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment](https://mlsanity.com/paper/2407.17777/) (2024)
- [Visual Question Answering on Multiple Remote Sensing Image Modalities](https://mlsanity.com/paper/2505.15401/) (2025)
- [FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping](https://mlsanity.com/paper/2506.07080/) (2025)
- [Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications](https://mlsanity.com/paper/2301.01200/) (2022)
- [LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing](https://mlsanity.com/paper/2404.13659/) (2024)
- [Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications](https://mlsanity.com/paper/2406.02601/) (2024)
- [Rainy: Unlocking Satellite Calibration for Deep Learning in Precipitation](https://mlsanity.com/paper/2504.10776/) (2025)
- [Cross-sensor self-supervised training and alignment for remote sensing](https://mlsanity.com/paper/2405.09922/) (2024)
- [CarbonSense: A Multimodal Dataset and Baseline for Carbon Flux Modelling](https://mlsanity.com/paper/2406.04940/) (2024)
- [Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View](https://mlsanity.com/paper/2406.05766/) (2024)
- [Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning](https://mlsanity.com/paper/2409.09670/) (2024)
- [The Moon's Many Faces: A Single Unified Transformer for Multimodal Lunar Reconstruction](https://mlsanity.com/paper/2505.05644/) (2025)
- [Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping](https://mlsanity.com/paper/2304.02407/) (2023)
- [Multi-Sensor Deep Learning for Glacier Mapping](https://mlsanity.com/paper/2409.12034/) (2024)

## Abstract (arXiv)

With the extremely rapid advances in remote sensing (RS) technology, a great quantity of Earth observation (EO) data featuring considerable and complicated heterogeneity is readily available nowadays, which renders researchers an opportunity to tackle current geoscience applications in a fresh way. With the joint utilization of EO data, much research on multimodal RS data fusion has made tremendous progress in recent years, yet these developed traditional algorithms inevitably meet the performance bottleneck due to the lack of the ability to comprehensively analyse and interpret these strongly heterogeneous data. Hence, this non-negligible limitation further arouses an intense demand for an alternative tool with powerful processing competence. Deep learning (DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in numerous computer vision tasks owing to its impressive ability in data representation and reconstruction. Naturally, it has been successfully applied to the field of multimodal RS data fusion, yielding great improvement compared with traditional methods. This survey aims to present a systematic overview in DL-based multimodal RS data fusion. More specifically, some essential knowledge about this topic is first given. Subsequently, a literature survey is conducted to analyse the trends of this field. Some prevalent sub-fields in the multimodal RS data fusion are then reviewed in terms of the to-be-fused data modalities, i.e., spatiospectral, spatiotemporal, light detection and ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data fusion. Furthermore, We collect and summarize some valuable resources for the sake of the development in multimodal RS data fusion. Finally, the remaining challenges and potential future directions are highlighted.