mlsanityData to 7 Oct 2026

Rationale-guided Prompting for Knowledge-based Visual Question Answering

arXiv 2412.16936 ↗First posted 22 Dec 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, Large Language Models (LLMs) have been used for knowledge-based Visual Question Answering (VQA). Despite the encouraging results of previous studies, prior methods prompt LLMs to predict answers directly, neglecting intermediate thought processes. We argue that prior methods do not sufficiently activate the capacities of LLMs. We propose a framework called PLRH that Prompts LLMs with Rationale Heuristics for knowledge-based VQA. The PLRH prompts LLMs with Chain of Thought (CoT) to generate rationale heuristics, i.e., intermediate thought processes, and then leverages the rationale heuristics to inspire LLMs to predict answers. Experiments show that our approach outperforms the existing baselines by more than 2.2 and 2.1 on OK-VQA and A-OKVQA, respectively.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

29 of 36 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
17 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  3. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  4. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  5. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  6. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  7. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  8. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  9. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  10. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoningcited by 60
  11. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  12. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  13. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  14. 2020Deep Multimodal Neural Architecture Searchcited by 34
  15. 2022Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignmentcited by 23
  16. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  17. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQAcited by 18

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 7.

9 papers.

  1. 2025Test-Time Learning for Large Language Modelscited by 46
  2. 2025Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Groundingcited by 17
  3. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 15
  4. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answeringcited by 10
  5. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  6. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  7. 2025EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Modelscited by 7
  8. 2024Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answeringcited by 3
  9. 2025LLMs are Introvertcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering
  2. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
  3. 2024Find The Gap: Knowledge Base Reasoning For Visual Question Answering
  4. 2023Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA
  5. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts
  6. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  7. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering
  8. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types