# Multi-Modal and Multi-Resolution Data Fusion for High-Resolution Cloud Removal: A Novel Baseline and Benchmark

arXiv: https://arxiv.org/abs/2301.03432 · HTML: https://mlsanity.com/paper/2301.03432/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 63 references that are in the map.
- First posted on arXiv 9 Jan 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [Residual Dense Network for Image Restoration](https://mlsanity.com/paper/1812.10477/) (2018)
- [Open High-Resolution Satellite Imagery: The WorldStrat Dataset -- With Application to Super-Resolution](https://mlsanity.com/paper/2207.06418/) (2022)
- [EarthNets: Empowering AI in Earth Observation](https://mlsanity.com/paper/2210.04936/) (2022)
- [Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery](https://mlsanity.com/paper/2009.07683/) (2020)
- [EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task](https://mlsanity.com/paper/2104.10066/) (2021)
- [SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal](https://mlsanity.com/paper/2201.09613/) (2022)
- [A Remote Sensing Image Dataset for Cloud Removal](https://mlsanity.com/paper/1901.00600/) (2019)
- [Cloud Removal for Remote Sensing Imagery via Spatial Attention Generative Adversarial Network](https://mlsanity.com/paper/2009.13015/) (2020)
- [Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1710.04835/) (2017)
- [GLF-CR: SAR-Enhanced Cloud Removal with Global-Local Fusion](https://mlsanity.com/paper/2206.02850/) (2022)
- [Cloud removal in remote sensing images using generative adversarial networks and SAR-to-optical image translation](https://mlsanity.com/paper/2012.12180/) (2020)
- [Cloud Removal in Satellite Images Using Spatiotemporal Generative Networks](https://mlsanity.com/paper/1912.06838/) (2019)
- [Multi-Head Linear Attention Generative Adversarial Network for Thin Cloud Removal](https://mlsanity.com/paper/2012.10898/) (2020)

## Built on by (most cited first, 7 of 7)

- [AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery](https://mlsanity.com/paper/2410.23891/) (2024)
- [CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding](https://mlsanity.com/paper/2602.12652/) (2026)
- [High-Quality Cloud-Free Optical Image Synthesis Using Multi-Temporal SAR and Contaminated Optical Data](https://mlsanity.com/paper/2504.16870/) (2025)
- [Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation](https://mlsanity.com/paper/2603.20811/) (2026)
- [Synthetic Aperture Radar Image Change Detection Based on Global Dynamic Context-Aware Network](https://mlsanity.com/paper/2605.16764/) (2026)
- [Deep Learning Super Resolution for Satellite Cloud Mask Downscaling](https://mlsanity.com/paper/2608.24715/) (2026)
- [OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation](https://mlsanity.com/paper/2610.15414/) (2026)

## Abstract (arXiv)

Cloud removal is a significant and challenging problem in remote sensing, and in recent years, there have been notable advancements in this area. However, two major issues remain hindering the development of cloud removal: the unavailability of high-resolution imagery for existing datasets and the absence of evaluation regarding the semantic meaningfulness of the generated structures. In this paper, we introduce M3R-CR, a benchmark dataset for high-resolution Cloud Removal with Multi-Modal and Multi-Resolution data fusion. With this dataset, we consider the problem of cloud removal in high-resolution optical remote sensing imagery by integrating multi-modal and multi-resolution information. In this context, we have to take into account the alignment errors caused by the multi-resolution nature, along with the more pronounced misalignment issues in high-resolution images due to inherent imaging mechanism differences and other factors. Existing multi-modal data fusion based methods, which assume the image pairs are aligned accurately at pixel-level, are thus not appropriate for this problem. To this end, we design a new baseline named Align-CR to perform the low-resolution SAR image guided high-resolution optical image cloud removal. It gradually warps and fuses the features of the multi-modal and multi-resolution data during the reconstruction process, effectively mitigating concerns associated with misalignment. In the experiments, we evaluate the performance of cloud removal by analyzing the quality of visually pleasing textures using image reconstruction metrics and further analyze the generation of semantically meaningful structures using a well-established semantic segmentation task. The proposed Align-CR method is superior to other baseline methods in both areas.