# EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation

arXiv: https://arxiv.org/abs/2605.29330 · HTML: https://mlsanity.com/paper/2605.29330/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Mission Critical -- Satellite Data is a Distinct Modality in Machine Learning](https://mlsanity.com/paper/2402.01444/) (2024)
- [Recent Advances in Domain Adaptation for the Classification of Remote Sensing Data](https://mlsanity.com/paper/2104.07778/) (2021)
- [reBEN: Refined BigEarthNet Dataset for Remote Sensing Image Analysis](https://mlsanity.com/paper/2407.03653/) (2024)
- [EarthNets: Empowering AI in Earth Observation](https://mlsanity.com/paper/2210.04936/) (2022)
- [TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation](https://mlsanity.com/paper/2506.06281/) (2025)
- [GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI](https://mlsanity.com/paper/2511.15658/) (2025)
- [Assessing out-of-domain generalization for robust building damage detection](https://mlsanity.com/paper/2011.10328/) (2020)
- [Multi-Region Transfer Learning for Segmentation of Crop Field Boundaries in Satellite Images with Limited Labels](https://mlsanity.com/paper/2404.00179/) (2024)
- [GeoCrossBench: Cross-Band Generalization for Remote Sensing](https://mlsanity.com/paper/2511.02831/) (2025)

## Built on by (most cited first, 11 of 11)

- [GEOID-Flood: A Large-Scale Multi-Modal Benchmark Dataset for Flood Segmentation](https://mlsanity.com/paper/2608.02315/) (2026)
- [Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts](https://mlsanity.com/paper/2608.16614/) (2026)
- [Curated retrieval versus open web search in public AI information services: a coverage-trust trade-off](https://mlsanity.com/paper/2607.05217/) (2026)
- [More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe](https://mlsanity.com/paper/2607.15942/) (2026)
- [Now We Know? A Systematic Comparison of TerraMind and THOR](https://mlsanity.com/paper/2607.18504/) (2026)
- [Earth Observation Foundation Models for Terrestrial Ecohydrology: From Representation Learning to Process Inference](https://mlsanity.com/paper/2608.15282/) (2026)
- [Quantifying geographic domain shift to decouple the geospatial transferability of human mobility flow generation models](https://mlsanity.com/paper/2608.21567/) (2026)
- [Infra-Bench CLS: A Global, Open-Source Benchmark for Critical Infrastructure Classification with Earth Observation Foundation Models](https://mlsanity.com/paper/2609.09482/) (2026)
- [RSPDBench: Benchmarking Vision Foundation Models on Earth Observation Tasks Under Physically Grounded Remote-Sensing Product Degradations](https://mlsanity.com/paper/2609.23427/) (2026)
- [FairRSFM: A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models](https://mlsanity.com/paper/2610.05790/) (2026)
- [Shortcut Learning in Geospatial Foundation Model Embeddings for Cross-Disaster Damage Assessment](https://mlsanity.com/paper/2610.13766/) (2026)

## Abstract (arXiv)

Current Earth observation benchmarks focus on measuring performance on diverse tasks and applications, typically measuring generalization in-distribution. But when models are deployed, they must generalize to myriad out-of-distribution scenarios, such as new time periods, geographies, scales, and sensors. We introduce EarthShift: the first public testbed for benchmarking robustness across multiple realistic distribution shifts encountered in remote sensing. EarthShift enables users to measure distributional robustness by comparing performance in- and out-of-distribution using datasets from paired datasets from different sources, temporal windows, geographic locations, and sensors. Our experiments on 8 geospatial foundation models (GFMs) and 11 tasks covering 5 shift types show that GFMs consistently perform 15-20% worse out-of-distribution on average regardless of model architecture, size, pre-training or fine-tuning strategy. We show that GFM robustness is similar to that of generic vision foundation models, and even fully-supervised models. This highlights a need for future research to strive for improvements in distributional robustness, not just performance, which can be benchmarked using EarthShift. We release our code and datasets to provide a testbed to guide future work to create foundation models that are robust and reliable in real-world applications. Code and data for EarthShift are available at: https://earthshift.github.io