# Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curation

arXiv: https://arxiv.org/abs/2504.06962 · HTML: https://mlsanity.com/paper/2504.06962/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 68 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Builds on (most cited first, 30 of 46)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery](https://mlsanity.com/paper/2312.10115/) (2023)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders](https://mlsanity.com/paper/2311.00566/) (2023)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [D4: Improving LLM Pretraining via Document De-Duplication and Diversification](https://mlsanity.com/paper/2308.12284/) (2023)
- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](https://mlsanity.com/paper/2412.02732/) (2024)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation](https://mlsanity.com/paper/2403.15356/) (2024)
- [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022)
- [Divide and Contrast: Self-supervised Learning from Uncurated Data](https://mlsanity.com/paper/2105.08054/) (2021)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [TorchGeo: Deep Learning With Geospatial Data](https://mlsanity.com/paper/2111.08872/) (2021)
- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [Vision Foundation Models in Remote Sensing: A Survey](https://mlsanity.com/paper/2408.03464/) (2024)
- [Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach](https://mlsanity.com/paper/2405.15613/) (2024)
- [Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need](https://mlsanity.com/paper/1901.11409/) (2019)
- [SpectralEarth: Training Hyperspectral Foundation Models at Scale](https://mlsanity.com/paper/2408.08447/) (2024)

## Built on by (most cited first, 7 of 7)

- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts](https://mlsanity.com/paper/2509.14104/) (2025)
- [MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data](https://mlsanity.com/paper/2508.10894/) (2025)
- [Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Need](https://mlsanity.com/paper/2507.04269/) (2025)
- [OceanSAR-2: A Universal Feature Extractor for SAR Ocean Observation](https://mlsanity.com/paper/2601.07392/) (2026)
- [Hide and Seek: Investigating Redundancy in Earth Observation Imagery](https://mlsanity.com/paper/2603.13524/) (2026)
- [Fidelity-Diversity-Consistency (FDC): Data Pruning for Remote Sensing Change Detection](https://mlsanity.com/paper/2608.21754/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has enabled the development of vision foundation models for Earth Observation (EO), demonstrating strong transferability across diverse remote sensing tasks. While prior work has focused on network architectures and training strategies, the role of dataset curation, especially in balancing and diversifying pre-training datasets, remains underexplored. In EO, this challenge is amplified by the redundancy and heavy-tailed distributions common in satellite imagery, which can lead to biased representations and inefficient training. In this work, we propose a dynamic dataset pruning strategy designed to improve SSL pre-training by maximizing dataset diversity and balance. Our method iteratively refines the training set without requiring a pre-existing feature extractor, making it well-suited for domains where curated datasets are limited or unavailable. We demonstrate our approach on the Sentinel-1 Wave Mode (WV) Synthetic Aperture Radar (SAR) archive, a challenging dataset dominated by ocean observations. We train models from scratch on the entire Sentinel-1 WV archive spanning 10 years. Across three downstream tasks, our results show that dynamic pruning improves both computational efficiency and representation quality, leading to stronger transferability. We also release the weights of OceanSAR-1, the first model in the OceanSAR family, a series of foundation models for ocean observation and analysis using SAR imagery, at github.com/galeio-research/OceanSAR-models/.