# Incomplete Multimodal Learning for Remote Sensing Data Fusion

arXiv: https://arxiv.org/abs/2304.11381 · HTML: https://mlsanity.com/paper/2304.11381/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 22 Apr 2023.

## Builds on (most cited first, 13 of 13)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [The effectiveness of MAE pre-pretraining for billion-scale pretraining](https://mlsanity.com/paper/2303.13496/) (2023)
- [Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks](https://mlsanity.com/paper/1711.08681/) (2017)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion](https://mlsanity.com/paper/1807.01569/) (2018)
- [Self-supervised Change Detection in Multi-view Remote Sensing Images](https://mlsanity.com/paper/2103.05969/) (2021)
- [Urban Land Cover Classification with Missing Data Modalities Using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.07383/) (2017)
- [Zorro: the masked multimodal transformer](https://mlsanity.com/paper/2301.09595/) (2023)
- [Overcoming Missing and Incomplete Modalities with Generative Adversarial Networks for Building Footprint Segmentation](https://mlsanity.com/paper/1808.03195/) (2018)
- [Road Extraction with Satellite Images and Partial Road Maps](https://mlsanity.com/paper/2303.12394/) (2023)

## Built on by (most cited first, 3 of 3)

- [XoFTR: Cross-modal Feature Matching Transformer](https://mlsanity.com/paper/2404.09692/) (2024)
- [Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2310.13876/) (2023)
- [From Pixels to Images: A Structural Survey of Deep Learning Paradigms in Remote Sensing Image Semantic Segmentation](https://mlsanity.com/paper/2505.15147/) (2025)

## Abstract (arXiv)

The mechanism of connecting multimodal signals through self-attention operation is a key factor in the success of multimodal Transformer networks in remote sensing data fusion tasks. However, traditional approaches assume access to all modalities during both training and inference, which can lead to severe degradation when dealing with modal-incomplete inputs in downstream applications. To address this limitation, our proposed approach introduces a novel model for incomplete multimodal learning in the context of remote sensing data fusion. This approach can be used in both supervised and self-supervised pretraining paradigms and leverages the additional learned fusion tokens in combination with Bi-LSTM attention and masked self-attention mechanisms to collect multimodal signals. The proposed approach employs reconstruction and contrastive loss to facilitate fusion in pre-training while allowing for random modality combinations as inputs in network training. Our approach delivers state-of-the-art performance on two multimodal datasets for tasks such as building instance / semantic segmentation and land-cover mapping tasks when dealing with incomplete inputs during inference.