# Multilingual Vision-Language Pre-training for the Remote Sensing Domain

arXiv: https://arxiv.org/abs/2410.23370 · HTML: https://mlsanity.com/paper/2410.23370/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 71 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://mlsanity.com/paper/2309.07915/) (2023)
- [SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference](https://mlsanity.com/paper/2312.01597/) (2023)
- [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [Open-Vocabulary Universal Image Segmentation with MaskCLIP](https://mlsanity.com/paper/2208.08984/) (2022)
- [Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2404.08181/) (2024)

## Built on by (most cited first, 7 of 7)

- [Domain Adaptation of VLM for Soccer Video Understanding](https://mlsanity.com/paper/2505.13860/) (2025)
- [FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping](https://mlsanity.com/paper/2506.07080/) (2025)
- [Multilingual Training-Free Remote Sensing Image Captioning](https://mlsanity.com/paper/2512.00887/) (2025)
- [An Efficient and Effective Encoder Model for Vision and Language Tasks in the Remote Sensing Domain](https://mlsanity.com/paper/2512.15531/) (2025)
- [One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensembling](https://mlsanity.com/paper/2606.08126/) (2026)
- [A Composition-Aware Pretraining Framework for Geospatial Foundation Models](https://mlsanity.com/paper/2608.30817/) (2026)
- [One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Data](https://mlsanity.com/paper/2609.26097/) (2026)

## Abstract (arXiv)

Methods based on Contrastive Language-Image Pre-training (CLIP) are nowadays extensively used in support of vision-and-language tasks involving remote sensing data, such as cross-modal retrieval. The adaptation of CLIP to this specific domain has relied on model fine-tuning with the standard contrastive objective, using existing human-labeled image-caption datasets, or using synthetic data corresponding to image-caption pairs derived from other annotations over remote sensing images (e.g., object classes). The use of different pre-training mechanisms has received less attention, and only a few exceptions have considered multilingual inputs. This work proposes a novel vision-and-language model for the remote sensing domain, exploring the fine-tuning of a multilingual CLIP model and testing the use of a self-supervised method based on aligning local and global representations from individual input images, together with the standard CLIP objective. Model training relied on assembling pre-existing datasets of remote sensing images paired with English captions, followed by the use of automated machine translation into nine additional languages. We show that translated data is indeed helpful, e.g. improving performance also on English. Our resulting model, which we named Remote Sensing Multilingual CLIP (RS-M-CLIP), obtains state-of-the-art results in a variety of vision-and-language tasks, including cross-modal and multilingual image-text retrieval, or zero-shot image classification.