# Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model

arXiv: https://arxiv.org/abs/2504.04988 · HTML: https://mlsanity.com/paper/2504.04988/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [RSVQA: Visual Question Answering for Remote Sensing Data](https://mlsanity.com/paper/2003.07333/) (2020)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain](https://mlsanity.com/paper/2401.16822/) (2024)
- [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing](https://mlsanity.com/paper/2312.12856/) (2023)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model](https://mlsanity.com/paper/2401.09712/) (2024)
- [Retrieval Augmented Classification for Long-Tail Visual Recognition](https://mlsanity.com/paper/2202.11233/) (2022)
- [Vision-Language Models in Remote Sensing: Current Progress and Future Trends](https://mlsanity.com/paper/2305.05726/) (2023)
- [VideoRAG: Retrieval-Augmented Generation over Video Corpus](https://mlsanity.com/paper/2501.05874/) (2025)
- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Models](https://mlsanity.com/paper/2402.11325/) (2024)
- [RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2411.01595/) (2024)
- [FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models](https://mlsanity.com/paper/2501.02461/) (2025)
- [Generalization-Enhanced Few-Shot Object Detection in Remote Sensing](https://mlsanity.com/paper/2501.02474/) (2025)
- [AutoEncoding Tree for City Generation and Applications](https://mlsanity.com/paper/2309.15941/) (2023)

## Built on by (most cited first, 6 of 6)

- [Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search](https://mlsanity.com/paper/2511.20460/) (2025)
- [Geospatial-Temporal Sensemaking of Remote Sensing Activity Detections with Multimodal Large Language Model](https://mlsanity.com/paper/2605.10739/) (2026)
- [Naiad: Novel Agentic Intelligent Autonomous System for Inland Water Monitoring](https://mlsanity.com/paper/2601.05256/) (2025)
- [GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs](https://mlsanity.com/paper/2608.00877/) (2026)
- [Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mapping](https://mlsanity.com/paper/2610.00302/) (2026)
- [TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2610.12517/) (2026)

## Abstract (arXiv)

Recent progress in VLMs has demonstrated impressive capabilities across a variety of tasks in the natural image domain. Motivated by these advancements, the remote sensing community has begun to adopt VLMs for remote sensing vision-language tasks, including scene understanding, image captioning, and visual question answering. However, existing remote sensing VLMs typically rely on closed-set scene understanding and focus on generic scene descriptions, yet lack the ability to incorporate external knowledge. This limitation hinders their capacity for semantic reasoning over complex or context-dependent queries that involve domain-specific or world knowledge. To address these challenges, we first introduced a multimodal Remote Sensing World Knowledge (RSWK) dataset, which comprises high-resolution satellite imagery and detailed textual descriptions for 14,141 well-known landmarks from 175 countries, integrating both remote sensing domain knowledge and broader world knowledge. Building upon this dataset, we proposed a novel Remote Sensing Retrieval-Augmented Generation (RS-RAG) framework, which consists of two key components. The Multi-Modal Knowledge Vector Database Construction module encodes remote sensing imagery and associated textual knowledge into a unified vector space. The Knowledge Retrieval and Response Generation module retrieves and re-ranks relevant knowledge based on image and/or text queries, and incorporates the retrieved content into a knowledge-augmented prompt to guide the VLM in producing contextually grounded responses. We validated the effectiveness of our approach on three representative vision-language tasks, including image captioning, image classification, and visual question answering, where RS-RAG significantly outperformed state-of-the-art baselines.