# SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal

arXiv: https://arxiv.org/abs/2201.09613 · HTML: https://mlsanity.com/paper/2201.09613/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [SEN12MS -- A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusion](https://mlsanity.com/paper/1906.07789/) (2019)
- [Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery](https://mlsanity.com/paper/2009.07683/) (2020)
- [An Internal Learning Approach to Video Inpainting](https://mlsanity.com/paper/1909.07957/) (2019)
- [Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1710.04835/) (2017)
- [Self-Supervised Multisensor Change Detection](https://mlsanity.com/paper/2103.05102/) (2021)
- [Weakly Supervised Semantic Segmentation of Satellite Images for Land Cover Mapping -- Challenges and Opportunities](https://mlsanity.com/paper/2002.08254/) (2020)
- [Remote Sensing Image Classification with the SEN12MS Dataset](https://mlsanity.com/paper/2104.00704/) (2021)

## Built on by (most cited first, 30 of 32)

- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [Major TOM: Expandable Datasets for Earth Observation](https://mlsanity.com/paper/2402.12095/) (2024)
- [UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2304.05464/) (2023)
- [DiffCR: A Fast Conditional Diffusion Framework for Cloud Removal from Optical Satellite Images](https://mlsanity.com/paper/2308.04417/) (2023)
- [GLF-CR: SAR-Enhanced Cloud Removal with Global-Local Fusion](https://mlsanity.com/paper/2206.02850/) (2022)
- [AI Security for Geoscience and Remote Sensing: Challenges and Future Trends](https://mlsanity.com/paper/2212.09360/) (2022)
- [Better, Not Just More: Data-Centric Machine Learning for Earth Observation](https://mlsanity.com/paper/2312.05327/) (2023)
- [Diffusion Enhancement for Cloud Removal in Ultra-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2401.15105/) (2024)
- [IDF-CR: Iterative Diffusion Process for Divide-and-Conquer Cloud Removal in Remote-sensing Images](https://mlsanity.com/paper/2403.11870/) (2024)
- [Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space](https://mlsanity.com/paper/2503.23717/) (2025)
- [Exploring the Capability of Text-to-Image Diffusion Models with Structural Edge Guidance for Multi-Spectral Satellite Image Inpainting](https://mlsanity.com/paper/2311.03008/) (2023)
- [U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2305.13277/) (2023)
- [Multi-Modal and Multi-Resolution Data Fusion for High-Resolution Cloud Removal: A Novel Baseline and Benchmark](https://mlsanity.com/paper/2301.03432/) (2023)
- [AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery](https://mlsanity.com/paper/2410.23891/) (2024)
- [Fus-MAE: A cross-attention-based data fusion approach for Masked Autoencoders in remote sensing](https://mlsanity.com/paper/2401.02764/) (2024)
- [Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications](https://mlsanity.com/paper/2301.01200/) (2022)
- [Attentive Contextual Attention for Cloud Removal](https://mlsanity.com/paper/2411.13042/) (2024)
- [DC4CR: When Cloud Removal Meets Diffusion Control in Remote Sensing](https://mlsanity.com/paper/2504.14785/) (2025)
- [Core-Set Selection for Data-efficient Land Cover Segmentation](https://mlsanity.com/paper/2505.01225/) (2025)
- [Detecting Looted Archaeological Sites from Satellite Image Time Series](https://mlsanity.com/paper/2409.09432/) (2024)
- [UniTS: Unified Spatio-Temporal Generative Model for Remote Sensing](https://mlsanity.com/paper/2512.04461/) (2025)
- [Benchmarking for Practice: Few-Shot Time-Series Crop-Type Classification on the EuroCropsML Dataset](https://mlsanity.com/paper/2504.11022/) (2025)
- [SADER: Structure-Aware Diffusion Framework with DEterministic Resampling for Multi-Temporal Remote Sensing Cloud Removal](https://mlsanity.com/paper/2602.00536/) (2026)
- [CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding](https://mlsanity.com/paper/2602.12652/) (2026)
- [Interpretation-Oriented Cloud Removal via Observation-Anchored Residual Flow with Geo-Contextual Alignment](https://mlsanity.com/paper/2607.02471/) (2026)
- [Difference-Driven Gating: Adaptive Feature Fusion for U-Net Decoder](https://mlsanity.com/paper/2607.11096/) (2026)
- [DehazeMamba: SAR-guided Optical Remote Sensing Image Dehazing with Adaptive State Space Model](https://mlsanity.com/paper/2503.13073/) (2025)
- [Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation](https://mlsanity.com/paper/2603.20811/) (2026)
- [Densification and forecasting of Sentinel-2 time series from multimodal SAR and Optical satellite data using deep generative models](https://mlsanity.com/paper/2605.04239/) (2026)
- [IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal](https://mlsanity.com/paper/2606.09347/) (2026)

## Abstract (arXiv)

About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote sensing practitioner's capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multi-modal and multi-temporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modal multi-temporal 3D-Convolution Neural Network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence-to-sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The conducted experiments highlight the contribution of our data set to the remote sensing community as well as the benefits of multi-modal and multi-temporal information to reconstruct noisy information. Our data set is available at https://patrickTUM.github.io/cloud_removal