mlsanityData to 14 Oct 2026

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

arXiv 2605.03790 ↗First posted 5 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

With advances in multimodal research and deep learning, Multimodal Large Language Models (MLLMs) have emerged as a powerful paradigm for a wide range of multimodal tasks. As a core problem in vision-language research, Visual Question Answering (VQA) has increasingly employed MLLMs to improve performance, particularly in open-domain settings where external knowledge is essential. In this work, we aim to further enhance retrieval-based VQA by more effectively integrating MLLMs with structured reasoning and knowledge acquisition. We introduce a logical prompting strategy that fuses Chain-of-Thought (CoT) reasoning with Visual Question Decomposition (VQD), termed CoVQD, to guide retrieval toward more accurate and relevant knowledge for MLLM inference. Building on this idea, we propose a new framework, CoVQD-guided RAG (CgRAG), which enables MLLMs to access more comprehensive and coherent external knowledge while benefiting from structured visual-text reasoning guidance, thereby improving generalization and reliability in complex cross-domain VQA scenarios. Extensive experiments on E-VQA, InfoSeek, and OKVQA benchmarks demonstrate the effectiveness of the proposed method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 59 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

46 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering16 Oct
  2. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework14 Apr
  3. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering25 Nov
  4. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models11 Oct
  5. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul
  6. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr

Other references · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
28 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  9. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
  10. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  11. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  12. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  13. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  14. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  15. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 71
  16. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  17. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  18. 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
  19. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 47
  20. 2023The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Modelscited by 22
  21. 2022Refined Semantic Enhancement towards Frequency Diffusion for Video Captioningcited by 18
  22. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  23. 2022REX: Reasoning-aware and Grounded Explanationcited by 12
  24. 2023Exploring Question Decomposition for Zero-Shot VQAcited by 12
  25. 2024Visual Question Decomposition on Multimodal Large Language Modelscited by 7
  26. 2023Logical Implications for Visual Question Answering Consistencycited by 6
  27. 2023Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQAcited by 5
  28. 2024Multimodal Rationales for Explainable Visual Question Answeringcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  2. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  3. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  4. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  5. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  6. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  7. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  8. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering