# Large Language Models for Captioning and Retrieving Remote Sensing Images

arXiv: https://arxiv.org/abs/2402.06475 · HTML: https://mlsanity.com/paper/2402.06475/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 80 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)

## Built on by (most cited first, 30 of 30)

- [MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation](https://mlsanity.com/paper/2405.13570/) (2024)
- [Towards Vision-Language Geo-Foundation Model: A Survey](https://mlsanity.com/paper/2406.09385/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesis](https://mlsanity.com/paper/2404.06637/) (2024)
- [Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)](https://mlsanity.com/paper/2503.11070/) (2025)
- [ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Models](https://mlsanity.com/paper/2402.11325/) (2024)
- [Multi-Agent Geospatial Copilots for Remote Sensing Workflows](https://mlsanity.com/paper/2501.16254/) (2025)
- [RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning](https://mlsanity.com/paper/2507.20776/) (2025)
- [A Survey on Remote Sensing Foundation Models: From Vision to Multimodality](https://mlsanity.com/paper/2503.22081/) (2025)
- [RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2411.01595/) (2024)
- [Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Models](https://mlsanity.com/paper/2503.00743/) (2025)
- [EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Prompting](https://mlsanity.com/paper/2504.12795/) (2025)
- [Knowledge-aware Text-Image Retrieval for Remote Sensing Images](https://mlsanity.com/paper/2405.03373/) (2024)
- [REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observation](https://mlsanity.com/paper/2412.16583/) (2024)
- [GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2602.14201/) (2026)
- [ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAG](https://mlsanity.com/paper/2411.07688/) (2024)
- [SSL4Eco: A Global Seasonal Dataset for Geospatial Foundation Models in Ecology](https://mlsanity.com/paper/2504.18256/) (2025)
- [From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing](https://mlsanity.com/paper/2411.05826/) (2024)
- [Survey of Multimodal Geospatial Foundation Models: Techniques, Applications, and Challenges](https://mlsanity.com/paper/2510.22964/) (2025)
- [Large Vision-Language Models for Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2411.10857/) (2024)
- [Semantic-Spatial Feature Fusion with Dynamic Graph Refinement for Remote Sensing Image Captioning](https://mlsanity.com/paper/2503.23453/) (2025)
- [WeaveEarth: Structured Evidence Construction and Reasoning for Training-Free UHR Remote Sensing Understanding](https://mlsanity.com/paper/2607.10120/) (2026)
- [Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques](https://mlsanity.com/paper/2410.17283/) (2024)
- [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025)
- [Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queries](https://mlsanity.com/paper/2512.14102/) (2025)
- [RAH-VLA: Resolution-Adaptive Hierarchical Vision-Language Alignment for Multimodal Remote Sensing Understanding](https://mlsanity.com/paper/2512.23243/) (2025)
- [Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding](https://mlsanity.com/paper/2604.11122/) (2026)
- [SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning](https://mlsanity.com/paper/2507.12845/) (2025)
- [Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?](https://mlsanity.com/paper/2607.20284/) (2026)
- [AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models](https://mlsanity.com/paper/2608.15456/) (2026)

## Abstract (arXiv)

Image captioning and cross-modal retrieval are examples of tasks that involve the joint analysis of visual and linguistic information. In connection to remote sensing imagery, these tasks can help non-expert users in extracting relevant Earth observation information for a variety of applications. Still, despite some previous efforts, the development and application of vision and language models to the remote sensing domain have been hindered by the relatively small size of the available datasets and models used in previous studies. In this work, we propose RS-CapRet, a Vision and Language method for remote sensing tasks, in particular image captioning and text-image retrieval. We specifically propose to use a highly capable large decoder language model together with image encoders adapted to remote sensing imagery through contrastive language-image pre-training. To bridge together the image encoder and language decoder, we propose training simple linear layers with examples from combining different remote sensing image captioning datasets, keeping the other parameters frozen. RS-CapRet can then generate descriptions for remote sensing images and retrieve images from textual descriptions, achieving SOTA or competitive performance with existing methods. Qualitative results illustrate that RS-CapRet can effectively leverage the pre-trained large language model to describe remote sensing images, retrieve them based on different types of queries, and also show the ability to process interleaved sequences of images and text in a dialogue manner.