# EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task

arXiv: https://arxiv.org/abs/2104.10066 · HTML: https://mlsanity.com/paper/2104.10066/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 74 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [SpaceNet: A Remote Sensing Dataset and Challenge Series](https://mlsanity.com/paper/1807.01232/) (2018)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://mlsanity.com/paper/1507.08750/) (2015)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere](https://mlsanity.com/paper/2003.11927/) (2020)
- [DeepSD: Generating High Resolution Climate Change Projections through Single Image Super-Resolution](https://mlsanity.com/paper/1703.03126/) (2017)
- [DenseNet: Implementing Efficient ConvNet Descriptor Pyramids](https://mlsanity.com/paper/1404.1869/) (2014)
- [Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis](https://mlsanity.com/paper/2001.01306/) (2020)
- [Disentangling Propagation and Generation for Video Prediction](https://mlsanity.com/paper/1812.00452/) (2018)
- [Assessing out-of-domain generalization for robust building damage detection](https://mlsanity.com/paper/2011.10328/) (2020)
- [Utilizing UNet for the future traffic map prediction task Traffic4cast challenge 2020](https://mlsanity.com/paper/2012.00125/) (2020)
- [Physics-informed GANs for Coastal Flood Visualization](https://mlsanity.com/paper/2010.08103/) (2020)
- [Estimating Crop Primary Productivity with Sentinel-2 and Landsat 8 using Machine Learning Methods Trained with Radiative Transfer Simulations](https://mlsanity.com/paper/2012.12101/) (2020)
- [Predicting Landscapes from Environmental Conditions Using Generative Networks](https://mlsanity.com/paper/1909.10296/) (2019)
- [PSIque: Next Sequence Prediction of Satellite Images using a Convolutional Sequence-to-Sequence Network](https://mlsanity.com/paper/1711.10644/) (2017)
- [High Resolution Forecasting of Heat Waves impacts on Leaf Area Index by Multiscale Multitemporal Deep Learning](https://mlsanity.com/paper/1909.07786/) (2019)

## Built on by (most cited first, 30 of 34)

- [DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation](https://mlsanity.com/paper/2203.12560/) (2022)
- [ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling](https://mlsanity.com/paper/2307.01909/) (2023)
- [EarthNets: Empowering AI in Earth Observation](https://mlsanity.com/paper/2210.04936/) (2022)
- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [ClimateSet: A Large-Scale Climate Model Dataset for Machine Learning](https://mlsanity.com/paper/2311.03721/) (2023)
- [GLF-CR: SAR-Enhanced Cloud Removal with Global-Local Fusion](https://mlsanity.com/paper/2206.02850/) (2022)
- [Multi-modal learning for geospatial vegetation forecasting](https://mlsanity.com/paper/2303.16198/) (2023)
- [U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2305.13277/) (2023)
- [Learning to forecast vegetation greenness at fine resolution over Africa with ConvLSTMs](https://mlsanity.com/paper/2210.13648/) (2022)
- [Multi-Modal and Multi-Resolution Data Fusion for High-Resolution Cloud Removal: A Novel Baseline and Benchmark](https://mlsanity.com/paper/2301.03432/) (2023)
- [Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network](https://mlsanity.com/paper/2406.13358/) (2024)
- [AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery](https://mlsanity.com/paper/2410.23891/) (2024)
- [A Diffusion-Based Framework for Terrain-Aware Remote Sensing Image Reconstruction](https://mlsanity.com/paper/2504.12112/) (2025)
- [Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications](https://mlsanity.com/paper/2301.01200/) (2022)
- [VegeDiff: Latent Diffusion Model for Geospatial Vegetation Forecasting](https://mlsanity.com/paper/2407.12592/) (2024)
- [ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method](https://mlsanity.com/paper/2504.07394/) (2025)
- [DeepExtremeCubes: Integrating Earth system spatio-temporal data for impact assessment of climate extremes](https://mlsanity.com/paper/2406.18179/) (2024)
- [Earth System Data Cubes: Avenues for advancing Earth system research](https://mlsanity.com/paper/2408.02348/) (2024)
- [Explainable Earth Surface Forecasting under Extreme Events](https://mlsanity.com/paper/2410.01770/) (2024)
- [An Analysis of Temporal Dropout in Earth Observation Time Series for Regression Tasks](https://mlsanity.com/paper/2504.06915/) (2025)
- [Deep Space Weather Model: Long-Range Solar Flare Prediction from Multi-Wavelength Images](https://mlsanity.com/paper/2508.07847/) (2025)
- [UniTS: Unified Spatio-Temporal Generative Model for Remote Sensing](https://mlsanity.com/paper/2512.04461/) (2025)
- [Implicit Neural Representations for Simultaneous Reduction and Continuous Reconstruction of Multi-Altitude Climate Data](https://mlsanity.com/paper/2409.17367/) (2024)
- [Identifying Spatio-Temporal Drivers of Extreme Events](https://mlsanity.com/paper/2410.24075/) (2024)
- [Video Compression for Spatiotemporal Earth System Data](https://mlsanity.com/paper/2506.19656/) (2025)
- [Cluster-Segregate-Perturb (CSP): A Model-agnostic Explainability Pipeline for Spatiotemporal Land Surface Forecasting Models](https://mlsanity.com/paper/2408.05916/) (2024)
- [Earth Science Foundation Models: From Perception to Reasoning and Discovery](https://mlsanity.com/paper/2605.12542/) (2026)
- [RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges](https://mlsanity.com/paper/2605.24945/) (2026)
- [HLS-GPT: A Generative Pretrained Transformer (GPT) for Continental-Scale NASA Harmonized Landsat and Sentinel-2 (HLS) Reflectance Reconstruction Across All Bands on Arbitrary Dates](https://mlsanity.com/paper/2606.18115/) (2026)
- [VegSim: A Geospatial World Model for Scenario-Conditioned Vegetation Simulation](https://mlsanity.com/paper/2606.21961/) (2026)

## Abstract (arXiv)

Satellite images are snapshots of the Earth surface. We propose to forecast them. We frame Earth surface forecasting as the task of predicting satellite imagery conditioned on future weather. EarthNet2021 is a large dataset suitable for training deep neural networks on the task. It contains Sentinel 2 satellite imagery at 20m resolution, matching topography and mesoscale (1.28km) meteorological variables packaged into 32000 samples. Additionally we frame EarthNet2021 as a challenge allowing for model intercomparison. Resulting forecasts will greatly improve (>x50) over the spatial resolution found in numerical models. This allows localized impacts from extreme weather to be predicted, thus supporting downstream applications such as crop yield prediction, forest health assessments or biodiversity monitoring. Find data, code, and how to participate at www.earthnet.tech