# Fewshot learning on global multimodal embeddings for earth observation tasks

arXiv: https://arxiv.org/abs/2310.00119 · HTML: https://mlsanity.com/paper/2310.00119/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model](https://mlsanity.com/paper/2208.03987/) (2022)
- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [Deep learning in remote sensing: a review](https://mlsanity.com/paper/1710.03959/) (2017)
- [Lightweight, Pre-trained Transformers for Remote Sensing Timeseries](https://mlsanity.com/paper/2304.14065/) (2023)
- [On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2304.06798/) (2023)
- [CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment](https://mlsanity.com/paper/2209.06430/) (2022)
- [SSL4EO-L: Datasets and Foundation Models for Landsat Imagery](https://mlsanity.com/paper/2306.09424/) (2023)
- [Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark](https://mlsanity.com/paper/2112.00570/) (2021)
- [Unlocking large-scale crop field delineation in smallholder farming systems with transfer learning and weak supervision](https://mlsanity.com/paper/2201.04771/) (2022)
- [Optical Remote Sensing Image Understanding with Weak Supervision: Concepts, Methods, and Perspectives](https://mlsanity.com/paper/2204.09120/) (2022)

## Built on by (most cited first, 6 of 6)

- [Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment](https://mlsanity.com/paper/2402.09816/) (2024)
- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Data](https://mlsanity.com/paper/2406.04230/) (2024)
- [On the Generalizability of Foundation Models for Crop Type Mapping](https://mlsanity.com/paper/2409.09451/) (2024)
- [Rapid Adaptation of Earth Observation Foundation Models for Segmentation](https://mlsanity.com/paper/2409.09907/) (2024)
- [Uncertainty and Generalizability in Foundation Models for Earth Observation](https://mlsanity.com/paper/2409.08744/) (2024)

## Abstract (arXiv)

In this work we pretrain a CLIP/ViT based model using three different modalities of satellite imagery across five AOIs covering over ~10% of Earth's total landmass, namely Sentinel 2 RGB optical imagery, Sentinel 1 SAR radar amplitude and interferometric coherence. This model uses ~ 250 M parameters. Then, we use the embeddings produced for each modality with a classical machine learning method to attempt different downstream tasks for earth observation related to vegetation, built up surface, croplands and permanent water. We consistently show how we reduce the need for labeled data by 99%, so that with ~200-500 randomly selected labeled examples (around 4K-10K km2) we reach performance levels analogous to those achieved with the full labeled datasets (about 150K image chips or 3M km2 in each area of interest - AOI) on all modalities, AOIs and downstream tasks. This leads us to think that the model has captured significant earth features useful in a wide variety of scenarios. To enhance our model's usability in practice, its architecture allows inference in contexts with missing modalities and even missing channels within each modality. Additionally, we visually show that this embedding space, obtained with no labels, is sensible to the different earth features represented by the labelled datasets we selected.