# Beyond the Visible: Multispectral Vision-Language Learning for Earth Observation

arXiv: https://arxiv.org/abs/2503.15969 · HTML: https://mlsanity.com/paper/2503.15969/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023)
- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](https://mlsanity.com/paper/2412.02732/) (2024)
- [SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing](https://mlsanity.com/paper/2312.12856/) (2023)
- [Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval](https://mlsanity.com/paper/2204.09868/) (2022)
- [Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment](https://mlsanity.com/paper/2312.06960/) (2023)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Models](https://mlsanity.com/paper/2402.11325/) (2024)
- [METER-ML: A Multi-Sensor Earth Observation Benchmark for Automated Methane Source Mapping](https://mlsanity.com/paper/2207.11166/) (2022)
- [SSL4EO-S12 v1.1: A Multimodal, Multiseasonal Dataset for Pretraining, Updated](https://mlsanity.com/paper/2503.00168/) (2025)

## Built on by (most cited first, 8 of 8)

- [TerraMind: Large-Scale Generative Multimodality for Earth Observation](https://mlsanity.com/paper/2504.11171/) (2025)
- [BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation](https://mlsanity.com/paper/2603.29630/) (2026)
- [TimeSenCLIP: A Time Series Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2508.11919/) (2025)
- [CLOSP: A Unified Semantic Space for SAR, MSI, and Text in Remote Sensing](https://mlsanity.com/paper/2507.10403/) (2025)
- [Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery](https://mlsanity.com/paper/2602.22613/) (2026)
- [Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning](https://mlsanity.com/paper/2604.21032/) (2026)
- [Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data](https://mlsanity.com/paper/2609.03391/) (2026)
- [MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation](https://mlsanity.com/paper/2610.18624/) (2026)

## Abstract (arXiv)

Vision-language models for Earth observation (EO) typically rely on the visual spectrum of data as the only model input, thus failing to leverage the rich spectral information available in the multispectral channels recorded by satellites. Therefore, we introduce Llama3-MS-CLIP, the first vision-language model pre-trained with contrastive learning on a large-scale multispectral dataset and report on the performance gains due to the extended spectral range. Furthermore, we present the largest-to-date image-caption dataset for multispectral data, consisting of one million Sentinel-2 samples and corresponding textual descriptions generated using Llama3-LLaVA-Next and Overture Maps data. We develop a scalable captioning pipeline, which is validated by domain experts. We evaluate Llama3-MS-CLIP on multispectral zero-shot image classification and retrieval using three datasets of varying complexity. Our results demonstrate that Llama3-MS-CLIP significantly outperforms other RGB-based approaches, improving classification accuracy by +6.77% on average and retrieval performance by +4.63% mAP compared to the second-best model. Our results emphasize the relevance of multispectral vision-language learning. The image-caption dataset, code, and model weights are available at https://github.com/IBM/MS-CLIP.