mlsanityData to 7 Oct 2026

AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning

arXiv 2604.17889 ↗First posted 20 Apr 2026cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Despite recent progress in multimodal large language models (MLLMs), reliable visual question answering in aerial scenes remains challenging. In such scenes, task-critical evidence is often carried by small objects, explicit quantities, coarse locations, and inter-object relations, whereas conventional dense visual-token representations are not well aligned with these structured semantics. To address this interface mismatch, we propose AeroRAG, a scene-graph-guided multimodal retrieval-augmented generation framework for visual question answering. The framework first converts an input image into structured visual knowledge, including object categories, quantities, spatial locations, and semantic relations, and then retrieves query-relevant semantic chunks to construct compact prompts for a text-based large language model. Rather than relying on direct reasoning over dense visual tokens, our method introduces a more explicit intermediate interface between perception and language reasoning. Experiments on the AUG aerial dataset and the general-domain VG-150 benchmark show consistent improvements over six strong MLLM baselines, with the largest gains observed in dense aerial scenes and relation-sensitive reasoning. We further evaluate the framework on VQAv2 to verify that the proposed interface remains compatible with standard visual reasoning settings. These results suggest that structured retrieval is a practical design direction for deployment-oriented and grounded visual reasoning systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 45 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

36 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representations20 May
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
22 more
  1. 2015Fast R-CNNcited by 2,930
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  9. 2023Multimodal Large Language Models: A Surveycited by 193
  10. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  11. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  12. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  13. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  14. 2023Prototype-based Embedding Network for Scene Graph Generationcited by 37
  15. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  16. 2023Adapter is All You Need for Tuning Visual Taskscited by 19
  17. 2025Towards Comprehensive Interactive Change Understanding in Remote Sensing: A Large-scale Dataset and Dual-granularity Enhanced VLMcited by 8
  18. 2024Towards Retrieval-Augmented Architectures for Image Captioningcited by 7
  19. 2024AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generationcited by 5
  20. 2025Scalable Audio-Visual Masked Autoencoders for Efficient Affective Video Facial Analysiscited by 5
  21. 2024Overcoming LLM Challenges using RAG-Driven Precision in Coffee Leaf Disease Remediationcited by 3
  22. 2025HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  2. 2025UR-Bench: A Benchmark for Multi-Hop Reasoning over Ultra-High-Resolution Images
  3. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  4. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
  5. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  6. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  7. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
  8. 2025MV-CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answering