mlsanityData to 14 Oct 2026

FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQA

arXiv 2502.18536 ↗First posted 25 Feb 2025cs.CV cs.CL cs.IRcited by 5

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Visual Question Answering requires models to generate accurate answers by integrating visual and textual understanding. However, VQA models still struggle with hallucinations, producing convincing but incorrect answers, particularly in knowledge-driven and Out-of-Distribution scenarios. We introduce FilterRAG, a retrieval-augmented framework that combines BLIP-VQA with Retrieval-Augmented Generation to ground answers in external knowledge sources like Wikipedia and DBpedia. FilterRAG achieves 36.5% accuracy on the OK-VQA dataset, demonstrating its effectiveness in reducing hallucinations and improving robustness in both in-domain and Out-of-Distribution settings. These findings highlight the potential of FilterRAG to improve Visual Question Answering systems for real-world deployment.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 50 of its 58 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

50 of 58 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  10. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  13. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  14. 2021A Survey on Automated Fact-Checkingcited by 378
  15. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  16. 2024An Introduction to Vision-Language Modelingcited by 178
  17. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  18. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  19. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  20. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  21. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  22. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  23. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Modelscited by 66
  24. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 59
  25. 2024LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioningcited by 59
  26. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  27. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  28. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  29. 2024MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalitiescited by 28
  30. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  31. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  32. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  33. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  34. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  35. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  36. 2013Joint Video and Text Parsing for Understanding Events and Answering Queriescited by 8
  37. 2022Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 7
  38. 2023General-Purpose Multi-Modal OOD Detection Frameworkcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  2. 2026OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolutioncited by 1
  3. 2026BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answeringcited by 1
  4. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMscited by 0
  5. 2026New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation
  2. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
  3. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge
  4. 2026WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata
  5. 2025GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucination
  6. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  7. 2022Retrieval Augmented Visual Question Answering with Outside Knowledge
  8. 2026Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting