# Re-ranking the Context for Multimodal Retrieval Augmented Generation

arXiv: https://arxiv.org/abs/2501.04695 · HTML: https://mlsanity.com/paper/2501.04695/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)
- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 5 of 5)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training](https://mlsanity.com/paper/2407.21439/) (2024)
- [Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images](https://mlsanity.com/paper/2311.14084/) (2023)
- [RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance](https://mlsanity.com/paper/2501.03995/) (2025)
- [Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation](https://mlsanity.com/paper/2408.01363/) (2024)

## Built on by (most cited first, 11 of 11)

- [Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs](https://mlsanity.com/paper/2510.17771/) (2025)
- [Multi-RAG: A Multimodal Retrieval-Augmented Generation System for Adaptive Video Understanding](https://mlsanity.com/paper/2505.23990/) (2025)
- [Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?](https://mlsanity.com/paper/2508.03644/) (2025)
- [VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.21556/) (2025)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [Very Efficient Listwise Multimodal Reranking for Long Documents](https://mlsanity.com/paper/2605.11864/) (2026)
- [Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment](https://mlsanity.com/paper/2506.10030/) (2025)
- [Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.15586/) (2025)
- [A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering](https://mlsanity.com/paper/2508.02841/) (2025)
- [Building Specialized Software-Assistant ChatBot with Graph-Based Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.05297/) (2025)
- [When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models](https://mlsanity.com/paper/2608.19208/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to generate RAG output. In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context from the knowledge-base in retrieval phase of the multi-modal RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance to select more relevant entries in retrieval process. The retrieval based on embeddings, say CLIP-based embedding, and cosine similarity usually perform poorly particularly for multi-modal data. We show that by using a more advanced relevancy measure, one can enhance the retrieval process by selecting more relevant pieces from the knowledge-base and eliminate the irrelevant pieces from the context by adaptively selecting up-to-k entries instead of fixed number of entries. Our evaluation using COCO dataset demonstrates significant enhancement in selecting relevant context and accuracy of the generated response.