mlsanityData to 14 Oct 2026

Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

arXiv 2411.16863 ↗First posted 25 Nov 2024cs.CV cs.AI cs.CLcited by 20

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal LLMs (MLLMs) are the natural extension of large language models to handle multimodal inputs, combining text and image data. They have recently garnered attention due to their capability to address complex tasks involving both modalities. However, their effectiveness is limited to the knowledge acquired during training, which restricts their practical utility. In this work, we introduce a novel method to enhance the adaptability of MLLMs by integrating external knowledge sources. Our proposed model, Reflective LLaVA (ReflectiVA), utilizes reflective tokens to dynamically determine the need for external knowledge and predict the relevance of information retrieved from an external database. Tokens are trained following a two-stage two-model training recipe. This ultimately enables the MLLM to manage external knowledge while preserving fluency and performance on tasks where external knowledge is not needed. Through our experiments, we demonstrate the efficacy of ReflectiVA for knowledge-based visual question answering, highlighting its superior performance compared to existing methods. Source code and trained models are publicly available at https://aimagelab.github.io/ReflectiVA.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 74 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

20 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models11 Oct
  2. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Mistral 7Bcited by 3,164
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2024Mixtral of Expertscited by 1,823
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
6 more
  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  2. 2024What matters when building vision-language models?cited by 326
  3. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  4. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  5. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  6. 2024CLAIR-A: Leveraging Large Language Models to Judge Audio Captionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  2. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  3. 2025Towards General Continuous Memory for Vision-Language Modelscited by 15
  4. 2026ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answeringcited by 5
  5. 2025From Symbolic to Neural and Back: Exploring Knowledge Graph-Large Language Model Synergiescited by 3
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  7. 2026CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigationcited by 2
  8. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
  9. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMscited by 1
  10. 2026REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignmentcited by 1
  11. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answeringcited by 1
  12. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
8 more
  1. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  2. 2025FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisationcited by 0
  3. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generationcited by 0
  4. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0
  5. 2026Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmentingcited by 0
  6. 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAGcited by 0
  7. 2026Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answeringcited by 0
  8. 2026SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
  2. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  3. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
  4. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  5. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
  6. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering
  7. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
  8. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering