# Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

arXiv: https://arxiv.org/abs/2411.16863 · HTML: https://mlsanity.com/paper/2411.16863/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters](https://mlsanity.com/paper/2402.04252/) (2024)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [EchoSight: Advancing Visual-Language Models with Wiki Knowledge](https://mlsanity.com/paper/2407.12735/) (2024)
- [RoRA-VLM: Robust Retrieval-Augmented Vision Language Models](https://mlsanity.com/paper/2410.08876/) (2024)
- [CLAIR-A: Leveraging Large Language Models to Judge Audio Captions](https://mlsanity.com/paper/2409.12962/) (2024)

## Built on by (most cited first, 20 of 20)

- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering](https://mlsanity.com/paper/2510.14605/) (2025)
- [Towards General Continuous Memory for Vision-Language Models](https://mlsanity.com/paper/2505.17670/) (2025)
- [ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2606.27974/) (2026)
- [From Symbolic to Neural and Back: Exploring Knowledge Graph-Large Language Model Synergies](https://mlsanity.com/paper/2506.09566/) (2025)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigation](https://mlsanity.com/paper/2606.28397/) (2026)
- [Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.22694/) (2025)
- [When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs](https://mlsanity.com/paper/2602.00344/) (2026)
- [REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignment](https://mlsanity.com/paper/2602.14065/) (2026)
- [MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2602.15915/) (2026)
- [MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.04969/) (2026)
- [MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems](https://mlsanity.com/paper/2605.19307/) (2026)
- [FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation](https://mlsanity.com/paper/2507.06622/) (2025)
- [Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.03790/) (2026)
- [Uncertainty Quantification for Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2605.29956/) (2026)
- [Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting](https://mlsanity.com/paper/2607.00159/) (2026)
- [Reason Before You Retrieve: Agentic Planning for Multi-modal RAG](https://mlsanity.com/paper/2607.22643/) (2026)
- [Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2608.21450/) (2026)
- [SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2608.21796/) (2026)

## Abstract (arXiv)

Multimodal LLMs (MLLMs) are the natural extension of large language models to handle multimodal inputs, combining text and image data. They have recently garnered attention due to their capability to address complex tasks involving both modalities. However, their effectiveness is limited to the knowledge acquired during training, which restricts their practical utility. In this work, we introduce a novel method to enhance the adaptability of MLLMs by integrating external knowledge sources. Our proposed model, Reflective LLaVA (ReflectiVA), utilizes reflective tokens to dynamically determine the need for external knowledge and predict the relevance of information retrieved from an external database. Tokens are trained following a two-stage two-model training recipe. This ultimately enables the MLLM to manage external knowledge while preserving fluency and performance on tasks where external knowledge is not needed. Through our experiments, we demonstrate the efficacy of ReflectiVA for knowledge-based visual question answering, highlighting its superior performance compared to existing methods. Source code and trained models are publicly available at https://aimagelab.github.io/ReflectiVA.