# Embedding Earth: Self-supervised contrastive pre-training for dense land cover classification

arXiv: https://arxiv.org/abs/2203.06041 · HTML: https://mlsanity.com/paper/2203.06041/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 63 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders](https://mlsanity.com/paper/1802.02080/) (2018)
- [Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis](https://mlsanity.com/paper/2001.01306/) (2020)
- [BreizhCrops: A Time Series Dataset for Crop Type Mapping](https://mlsanity.com/paper/1905.11893/) (2019)

## Built on by (most cited first, 6 of 6)

- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [ViTs for SITS: Vision Transformers for Satellite Image Time Series](https://mlsanity.com/paper/2301.04944/) (2023)
- [PhilEO Bench: Evaluating Geo-Spatial Foundation Models](https://mlsanity.com/paper/2401.04464/) (2024)
- [Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection](https://mlsanity.com/paper/2205.13769/) (2022)
- [Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space](https://mlsanity.com/paper/2412.05600/) (2024)
- [SatSwinMAE: Efficient Autoencoding for Multiscale Time-series Satellite Imagery](https://mlsanity.com/paper/2405.02512/) (2024)

## Abstract (arXiv)

In training machine learning models for land cover semantic segmentation there is a stark contrast between the availability of satellite imagery to be used as inputs and ground truth data to enable supervised learning. While thousands of new satellite images become freely available on a daily basis, getting ground truth data is still very challenging, time consuming and costly. In this paper we present Embedding Earth a self-supervised contrastive pre-training method for leveraging the large availability of satellite imagery to improve performance on downstream dense land cover classification tasks. Performing an extensive experimental evaluation spanning four countries and two continents we use models pre-trained with our proposed method as initialization points for supervised land cover semantic segmentation and observe significant improvements up to 25% absolute mIoU. In every case tested we outperform random initialization, especially so when ground truth data are scarse. Through a series of ablation studies we explore the qualities of the proposed approach and find that learnt features can generalize between disparate regions opening up the possibility of using the proposed pre-training scheme as a replacement to random initialization for Earth observation tasks. Code will be uploaded soon at https://github.com/michaeltrs/DeepSatModels.