# Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation

arXiv: https://arxiv.org/abs/2403.15356 · HTML: https://mlsanity.com/paper/2403.15356/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 102 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)

## Built on by (most cited first, 30 of 104)

- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](https://mlsanity.com/paper/2412.02732/) (2024)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [On the Foundations of Earth and Climate Foundation Models](https://mlsanity.com/paper/2405.04285/) (2024)
- [Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining](https://mlsanity.com/paper/2405.20462/) (2024)
- [LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation](https://mlsanity.com/paper/2411.09301/) (2024)
- [FlexiMo: A Flexible Remote Sensing Foundation Model](https://mlsanity.com/paper/2503.23844/) (2025)
- [REOBench: Benchmarking Robustness of Earth Observation Foundation Models](https://mlsanity.com/paper/2505.16793/) (2025)
- [No One Knows the State of the Art in Geospatial Foundation Models](https://mlsanity.com/paper/2605.12678/) (2026)
- [A2-MAE: A spatial-temporal-spectral unified remote sensing pre-training method based on anchor-aware masked autoencoder](https://mlsanity.com/paper/2406.08079/) (2024)
- [SeaMo: A Season-Aware Multimodal Foundation Model for Remote Sensing](https://mlsanity.com/paper/2412.19237/) (2024)
- [Geospatial Foundation Models to Enable Progress on Sustainable Development Goals](https://mlsanity.com/paper/2505.24528/) (2025)
- [SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images](https://mlsanity.com/paper/2506.19585/) (2025)
- [Harvesting AlphaEarth: Benchmarking the Geospatial Foundation Model for Agricultural Downstream Tasks](https://mlsanity.com/paper/2601.00857/) (2025)
- [OpenEarth-Agent: From Tool Calling to Tool Creation for Open-Environment Earth Observation](https://mlsanity.com/paper/2603.22148/) (2026)
- [RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs](https://mlsanity.com/paper/2604.07765/) (2026)
- [CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation](https://mlsanity.com/paper/2405.01217/) (2024)
- [Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curation](https://mlsanity.com/paper/2504.06962/) (2025)
- [NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observation](https://mlsanity.com/paper/2510.17914/) (2025)
- [Paving the way toward foundation models for irregular and unaligned Satellite Image Time Series](https://mlsanity.com/paper/2407.08448/) (2024)
- [SatMamba: Development of Foundation Models for Remote Sensing Imagery Using State Space Models](https://mlsanity.com/paper/2502.00435/) (2025)
- [A Survey on SAR ship classification using Deep Learning](https://mlsanity.com/paper/2503.11906/) (2025)
- [SpectralX: Parameter-efficient Domain Generalization for Spectral Remote Sensing Foundation Models](https://mlsanity.com/paper/2508.01731/) (2025)
- [Earth Embeddings as Products: Taxonomy, Ecosystem, and Standardized Access](https://mlsanity.com/paper/2601.13134/) (2026)
- [TESSERA v2: Scaling Pixel-wise Earth Foundation Models](https://mlsanity.com/paper/2607.03949/) (2026)
- [EarthScape: A Multimodal Dataset for Surficial Geologic Mapping and Earth Surface Analysis](https://mlsanity.com/paper/2503.15625/) (2025)
- [A Genealogy of Foundation Models in Remote Sensing](https://mlsanity.com/paper/2504.17177/) (2025)
- [GeoCrossBench: Cross-Band Generalization for Remote Sensing](https://mlsanity.com/paper/2511.02831/) (2025)
- [Transforming Weather Data from Pixel to Latent Space](https://mlsanity.com/paper/2503.06623/) (2025)
- [SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images](https://mlsanity.com/paper/2508.05202/) (2025)
- [TerraMAE: Learning Spatial-Spectral Representations from Hyperspectral Earth Observation Data via Adaptive Masked Autoencoders](https://mlsanity.com/paper/2508.07020/) (2025)

## Abstract (arXiv)

Earth observation (EO) in open-world settings presents a unique challenge: different applications rely on diverse sensor modalities, each with varying ground sampling distances, spectral ranges, and numbers of spectral bands. However, existing EO foundation models are typically tailored to specific sensor types, making them inflexible when generalizing across the heterogeneous landscape of EO data. To address this, we propose the Dynamic One-For-All (DOFA) model, a unified, multimodal foundation framework designed for diverse vision tasks in EO. Inspired by neural plasticity, DOFA utilizes a wavelength-conditioned dynamic hypernetwork to process inputs from five distinct satellite sensors flexibly. By continually pretraining on five EO modalities, DOFA achieves state-of-the-art performance across multiple downstream tasks and generalizes well to unseen modalities. Enhanced with hybrid continual pretraining, DOFA+ requires significantly fewer computational resources while outperforming counterparts trained with extensive GPU budgets. Experiments on diverse datasets highlight DOFA's potential as a foundation for general-purpose vision models in the sensor-diverse EO domain. The code and pre-trained weights are publicly available at https://github.com/zhu-xlab/DOFA.