mlsanityData to 7 Oct 2026

Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering

arXiv 2602.06050 ↗First posted 14 Jan 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Despite the remarkable capabilities of Large Vision Language Models (LVLMs), they still lack detailed knowledge about specific entities. Retrieval-augmented Generation (RAG) is a widely adopted solution that enhances LVLMs by providing additional contexts from an external Knowledge Base. However, we observe that previous decoding methods for RAG are sub-optimal as they fail to sufficiently leverage multiple relevant contexts and suppress the negative effects of irrelevant contexts. To this end, we propose Relevance-aware Multi-context Contrastive Decoding (RMCD), a novel decoding method for RAG. RMCD outputs a final prediction by combining outputs predicted with each context, where each output is weighted based on its relevance to the question. By doing so, RMCD effectively aggregates useful information from multiple relevant contexts while also counteracting the negative effects of irrelevant ones. Experiments show that RMCD consistently outperforms other decoding methods across multiple LVLMs, achieving the best performance on three knowledge-intensive visual question-answering benchmarks. Also, RMCD can be simply applied by replacing the decoding method of LVLMs without additional training. Analyses also show that RMCD is robust to the retrieval results, consistently performing the best across the weakest to the strongest retrieval results. Code is available at https://github.com/mlvlab/RMCD.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 63 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

52 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrieval31 Jul
  2. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr
  3. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
37 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  13. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  14. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  15. 2018nocaps: novel object captioning at scalecited by 444
  16. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  17. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  18. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  19. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  20. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  21. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  22. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  23. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  24. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  25. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  26. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  27. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  28. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  29. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  30. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  31. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  32. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  33. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  34. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  35. 2025VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioningcited by 9
  36. 2024Generative Subgraph Retrieval for Knowledge Graph-Grounded Dialog Generationcited by 6
  37. 2025TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  2. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  3. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
  4. 2026KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
  5. 2025Solution for Meta KDD Cup'25: A Comprehensive Three-Step Framework for Vision Question Answering ↗
  6. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  7. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  8. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation