# U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series

arXiv: https://arxiv.org/abs/2305.13277 · HTML: https://mlsanity.com/paper/2305.13277/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 88 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [A high-resolution canopy height model of the Earth](https://mlsanity.com/paper/2204.08322/) (2022)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [ViTs for SITS: Vision Transformers for Satellite Image Time Series](https://mlsanity.com/paper/2301.04944/) (2023)
- [Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery](https://mlsanity.com/paper/2009.07683/) (2020)
- [EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task](https://mlsanity.com/paper/2104.10066/) (2021)
- [An Internal Learning Approach to Video Inpainting](https://mlsanity.com/paper/1909.07957/) (2019)
- [SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal](https://mlsanity.com/paper/2201.09613/) (2022)
- [Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1710.04835/) (2017)
- [GLF-CR: SAR-Enhanced Cloud Removal with Global-Local Fusion](https://mlsanity.com/paper/2206.02850/) (2022)
- [Gating Revisited: Deep Multi-layer RNNs That Can Be Trained](https://mlsanity.com/paper/1911.11033/) (2019)
- [Seeing Through Clouds in Satellite Images](https://mlsanity.com/paper/2106.08408/) (2021)
- [Crop Classification under Varying Cloud Cover with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2012.02542/) (2020)

## Built on by (most cited first, 8 of 8)

- [Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network](https://mlsanity.com/paper/2406.13358/) (2024)
- [AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery](https://mlsanity.com/paper/2410.23891/) (2024)
- [UniTS: Unified Spatio-Temporal Generative Model for Remote Sensing](https://mlsanity.com/paper/2512.04461/) (2025)
- [CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration](https://mlsanity.com/paper/2607.27898/) (2026)
- [A Joint Learning Framework with Feature Reconstruction and Prediction for Incomplete Satellite Image Time Series in Agricultural Semantic Segmentation](https://mlsanity.com/paper/2505.19159/) (2025)
- [Asynchronous Remote Sensing Time-Series Fusion for Cloud Removal and Anytime Reconstruction](https://mlsanity.com/paper/2605.27726/) (2026)
- [Densification and forecasting of Sentinel-2 time series from multimodal SAR and Optical satellite data using deep generative models](https://mlsanity.com/paper/2605.04239/) (2026)
- [HLS-GPT: A Generative Pretrained Transformer (GPT) for Continental-Scale NASA Harmonized Landsat and Sentinel-2 (HLS) Reflectance Reconstruction Across All Bands on Arbitrary Dates](https://mlsanity.com/paper/2606.18115/) (2026)

## Abstract (arXiv)

Satellite image time series in the optical and infrared spectrum suffer from frequent data gaps due to cloud cover, cloud shadows, and temporary sensor outages. It has been a long-standing problem of remote sensing research how to best reconstruct the missing pixel values and obtain complete, cloud-free image sequences. We approach that problem from the perspective of representation learning and develop U-TILISE, an efficient neural model that is able to implicitly capture spatio-temporal patterns of the spectral intensities, and that can therefore be trained to map a cloud-masked input sequence to a cloud-free output sequence. The model consists of a convolutional spatial encoder that maps each individual frame of the input sequence to a latent encoding; an attention-based temporal encoder that captures dependencies between those per-frame encodings and lets them exchange information along the time dimension; and a convolutional spatial decoder that decodes the latent embeddings back into multi-spectral images. We experimentally evaluate the proposed model on EarthNet2021, a dataset of Sentinel-2 time series acquired all over Europe, and demonstrate its superior ability to reconstruct the missing pixels. Compared to a standard interpolation baseline, it increases the PSNR by 1.8 dB at previously seen locations and by 1.3 dB at unseen locations.