# TerraMind: Large-Scale Generative Multimodality for Earth Observation

arXiv: https://arxiv.org/abs/2504.11171 · HTML: https://mlsanity.com/paper/2504.11171/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 75 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark](https://mlsanity.com/paper/1909.00133/) (2019)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders](https://mlsanity.com/paper/2311.00566/) (2023)
- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [DiffusionSat: A Generative Foundation Model for Satellite Imagery](https://mlsanity.com/paper/2312.03606/) (2023)
- [Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation](https://mlsanity.com/paper/2303.11851/) (2023)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2005.07310/) (2020)
- [MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning](https://mlsanity.com/paper/2405.02771/) (2024)
- [Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review](https://mlsanity.com/paper/2205.01380/) (2022)
- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [Bridging Remote Sensors with Multisensor Geospatial Foundation Models](https://mlsanity.com/paper/2404.01260/) (2024)
- [MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation](https://mlsanity.com/paper/2405.13570/) (2024)
- [FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery](https://mlsanity.com/paper/2310.13336/) (2023)
- [Major TOM: Expandable Datasets for Earth Observation](https://mlsanity.com/paper/2402.12095/) (2024)
- [Self-supervised Change Detection in Multi-view Remote Sensing Images](https://mlsanity.com/paper/2103.05969/) (2021)
- [xView3-SAR: Detecting Dark Fishing Activity Using Synthetic Aperture Radar Imagery](https://mlsanity.com/paper/2206.00897/) (2022)
- [PhilEO Bench: Evaluating Geo-Spatial Foundation Models](https://mlsanity.com/paper/2401.04464/) (2024)
- [Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification](https://mlsanity.com/paper/2207.07189/) (2022)

## Built on by (most cited first, 30 of 104)

- [SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images](https://mlsanity.com/paper/2506.19585/) (2025)
- [THOR: A Versatile Foundation Model for Earth Observation Climate and Society Applications](https://mlsanity.com/paper/2601.16011/) (2026)
- [MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training](https://mlsanity.com/paper/2602.06285/) (2026)
- [FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping](https://mlsanity.com/paper/2506.07080/) (2025)
- [Surya: Foundation Model for Heliophysics](https://mlsanity.com/paper/2508.14112/) (2025)
- [Scalable Geospatial Data Generation Using AlphaEarth Foundations Model](https://mlsanity.com/paper/2508.11739/) (2025)
- [Earth Embeddings as Products: Taxonomy, Ecosystem, and Standardized Access](https://mlsanity.com/paper/2601.13134/) (2026)
- [PRUE: A Practical Recipe for Field Boundary Segmentation at Scale](https://mlsanity.com/paper/2603.27101/) (2026)
- [UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation](https://mlsanity.com/paper/2606.23503/) (2026)
- [TESSERA v2: Scaling Pixel-wise Earth Foundation Models](https://mlsanity.com/paper/2607.03949/) (2026)
- [GeoCrossBench: Cross-Band Generalization for Remote Sensing](https://mlsanity.com/paper/2511.02831/) (2025)
- [RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation](https://mlsanity.com/paper/2512.05025/) (2025)
- [The Moon's Many Faces: A Single Unified Transformer for Multimodal Lunar Reconstruction](https://mlsanity.com/paper/2505.05644/) (2025)
- [AQUAH: Automatic Quantification and Unified Agent in Hydrology](https://mlsanity.com/paper/2508.02936/) (2025)
- [CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts](https://mlsanity.com/paper/2509.14104/) (2025)
- [Leveraging Compact Satellite Embeddings and Graph Neural Networks for Large-Scale Poverty Mapping](https://mlsanity.com/paper/2511.01408/) (2025)
- [SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation](https://mlsanity.com/paper/2511.10370/) (2025)
- [CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis](https://mlsanity.com/paper/2504.19223/) (2025)
- [MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data](https://mlsanity.com/paper/2508.10894/) (2025)
- [A Sentinel-3 foundation model for ocean colour](https://mlsanity.com/paper/2509.21273/) (2025)
- [EO-VAE: Towards A Multi-sensor Tokenizer for Earth Observation Data](https://mlsanity.com/paper/2602.12177/) (2026)
- [TerraFlow: Multimodal, Multitemporal Representation Learning for Earth Observation](https://mlsanity.com/paper/2603.12762/) (2026)
- [SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation](https://mlsanity.com/paper/2603.19873/) (2026)
- [GeoSANE: Learning Geospatial Representations from Models, Not Data](https://mlsanity.com/paper/2603.23408/) (2026)
- [Location Is All You Need: Continuous Spatiotemporal Neural Representations of Earth Observation Data](https://mlsanity.com/paper/2604.07092/) (2026)
- [Agentic AI for Remote Sensing: Technical Challenges and Research Directions](https://mlsanity.com/paper/2604.24919/) (2026)
- [SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining](https://mlsanity.com/paper/2605.21075/) (2026)
- [Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing](https://mlsanity.com/paper/2607.03644/) (2026)
- [SLED: Scalable Location Encoding via Distillation](https://mlsanity.com/paper/2608.06612/) (2026)
- [PolyMicros: Bootstrapping a Foundation Model for Polycrystalline Material Structure](https://mlsanity.com/paper/2506.11055/) (2025)

## Abstract (arXiv)

We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO). Unlike other multimodal models, TerraMind is pretrained on dual-scale representations combining both token-level and pixel-level data across modalities. On a token level, TerraMind encodes high-level contextual information to learn cross-modal relationships, while on a pixel level, TerraMind leverages fine-grained representations to capture critical spatial nuances. We pretrained TerraMind on nine geospatial modalities of a global, large-scale dataset. In this paper, we demonstrate that (i) TerraMind's dual-scale early fusion approach unlocks a range of zero-shot and few-shot applications for Earth observation, (ii) TerraMind introduces "Thinking-in-Modalities" (TiM) -- the capability of generating additional artificial data during finetuning and inference to improve the model output -- and (iii) TerraMind achieves beyond state-of-the-art performance in community-standard benchmarks for EO like PANGAEA. The pretraining dataset, the model weights, and our code are open-sourced under a permissive license.