# MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning

arXiv: https://arxiv.org/abs/2405.02771 · HTML: https://mlsanity.com/paper/2405.02771/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 60 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [Geography-Aware Self-Supervised Learning](https://mlsanity.com/paper/2011.09980/) (2020)
- [Benchmarking Representation Learning for Natural World Image Collections](https://mlsanity.com/paper/2103.16483/) (2021)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)

## Built on by (most cited first, 30 of 77)

- [TerraMind: Large-Scale Generative Multimodality for Earth Observation](https://mlsanity.com/paper/2504.11171/) (2025)
- [HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model](https://mlsanity.com/paper/2406.11519/) (2024)
- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [Towards a Unified Copernicus Foundation Model for Earth Vision](https://mlsanity.com/paper/2503.11849/) (2025)
- [TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis](https://mlsanity.com/paper/2506.20380/) (2025)
- [Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation](https://mlsanity.com/paper/2503.10845/) (2025)
- [A Survey on Remote Sensing Foundation Models: From Vision to Multimodality](https://mlsanity.com/paper/2503.22081/) (2025)
- [TerraMesh: A Planetary Mosaic of Multimodal Earth Observation Data](https://mlsanity.com/paper/2504.11172/) (2025)
- [WildSAT: Learning Satellite Image Representations from Wildlife Observations](https://mlsanity.com/paper/2412.14428/) (2024)
- [Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space](https://mlsanity.com/paper/2412.05600/) (2024)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)
- [No One Knows the State of the Art in Geospatial Foundation Models](https://mlsanity.com/paper/2605.12678/) (2026)
- [RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing](https://mlsanity.com/paper/2503.10392/) (2025)
- [GAIR: Location-Aware Self-Supervised Contrastive Pre-Training with Geo-Aligned Implicit Representations](https://mlsanity.com/paper/2503.16683/) (2025)
- [Geospatial Foundation Models to Enable Progress on Sustainable Development Goals](https://mlsanity.com/paper/2505.24528/) (2025)
- [Measuring the Intrinsic Dimension of Earth Representations](https://mlsanity.com/paper/2511.02101/) (2025)
- [SSL4EO-S12 v1.1: A Multimodal, Multiseasonal Dataset for Pretraining, Updated](https://mlsanity.com/paper/2503.00168/) (2025)
- [SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images](https://mlsanity.com/paper/2506.19585/) (2025)
- [From Physics to Foundation Models: A Review of AI-Driven Quantitative Remote Sensing Inversion](https://mlsanity.com/paper/2507.09081/) (2025)
- [GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data](https://mlsanity.com/paper/2509.26016/) (2025)
- [THOR: A Versatile Foundation Model for Earth Observation Climate and Society Applications](https://mlsanity.com/paper/2601.16011/) (2026)
- [MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training](https://mlsanity.com/paper/2602.06285/) (2026)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?](https://mlsanity.com/paper/2501.12535/) (2025)
- [Ice-FMBench: A Foundation Model Benchmark for Sea Ice Type Segmentation](https://mlsanity.com/paper/2503.22516/) (2025)
- [SSL4Eco: A Global Seasonal Dataset for Geospatial Foundation Models in Ecology](https://mlsanity.com/paper/2504.18256/) (2025)
- [MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks](https://mlsanity.com/paper/2505.14951/) (2025)
- [Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery](https://mlsanity.com/paper/2507.13385/) (2025)
- [Cropland Mapping using Geospatial Embeddings](https://mlsanity.com/paper/2511.02923/) (2025)
- [Earth Embeddings as Products: Taxonomy, Ecosystem, and Standardized Access](https://mlsanity.com/paper/2601.13134/) (2026)

## Abstract (arXiv)

The volume of unlabelled Earth observation (EO) data is huge, but many important applications lack labelled training data. However, EO data offers the unique opportunity to pair data from different modalities and sensors automatically based on geographic location and time, at virtually no human labor cost. We seize this opportunity to create MMEarth, a diverse multi-modal pretraining dataset at global scale. Using this new corpus of 1.2 million locations, we propose a Multi-Pretext Masked Autoencoder (MP-MAE) approach to learn general-purpose representations for optical satellite images. Our approach builds on the ConvNeXt V2 architecture, a fully convolutional masked autoencoder (MAE). Drawing upon a suite of multi-modal pretext tasks, we demonstrate that our MP-MAE approach outperforms both MAEs pretrained on ImageNet and MAEs pretrained on domain-specific satellite images. This is shown on several downstream tasks including image classification and semantic segmentation. We find that pretraining with multi-modal pretext tasks notably improves the linear probing performance compared to pretraining on optical satellite images only. This also leads to better label efficiency and parameter efficiency which are crucial aspects in global scale applications.