Data to 7 Oct 2026

MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training

arXiv 2407.21439 ↗First posted 31 Jul 2024cs.AI cs.CL cs.LGcited by 50

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more

Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in processing and generating content across multiple data modalities. However, a significant drawback of MLLMs is their reliance on static training data, leading to outdated information and limited contextual awareness. This static nature hampers their ability to provide accurate and up-to-date responses, particularly in dynamic or rapidly evolving contexts. Though integrating Multimodal Retrieval-augmented Generation (Multimodal RAG) offers a promising solution, the system would inevitably encounter the multi-granularity noisy correspondence (MNC) problem, which hinders accurate retrieval and generation. In this work, we propose RagVL, a novel framework with knowledge-enhanced reranking and noise-injected training, to address these limitations. We instruction-tune the MLLM with a simple yet effective instruction template to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. For generation, we inject visual noise during training at the data and token levels to enhance the generator's robustness. Extensive experiments on the subsets of two datasets that require retrieving and reasoning over images to answer a given query verify the effectiveness of our method. Code and models are available at https://github.com/IDEA-FinAI/RagVL.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 49 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

43 of 49 references.

References · 43

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2023GPT-4 Technical Reportcited by 18,132
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2023Visual Instruction Tuningcited by 7,792
2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690

31 more

2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2023Improved Baselines with Visual Instruction Tuningcited by 4,362
2021Zero-Shot Text-to-Image Generationcited by 4,276
2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
2021Improving language models by retrieving from trillions of tokenscited by 1,278
2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
2018Stacked Cross Attention for Image-Text Matchingcited by 414
2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
2021WebQA: Multihop and Multimodal QAcited by 109
2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
2022Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generationcited by 28
2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignmentcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 23; 2026: 22.

50 papers.

Sort:

By area:

2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20

38 more

2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
2025Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection?cited by 17
2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
2025Re-ranking the Context for Multimodal Retrieval Augmented Generationcited by 11
2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agentscited by 7
2025Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encodingcited by 4
2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
2025MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondencecited by 3
2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoningcited by 2
2025How to make Medical AI Systems safer? Simulating Vulnerabilities, and Threats in Multimodal Medical RAG Systemcited by 2
2026V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrievalcited by 2
2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 2
2026MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAGcited by 2
2026EviRank: Structured Relevance Evidence for Multimodal Image Re-rankingcited by 2
2025A Visual RAG Pipeline for Few-Shot Fine-Grained Product Classificationcited by 1
2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
2026POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Managementcited by 1
2026DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agentscited by 1
2026Very Efficient Listwise Multimodal Reranking for Long Documentscited by 1
2026UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrievalcited by 1
2026GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memorycited by 1
2025VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samplescited by 0
2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generationcited by 0
2026UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidatescited by 0
2026Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generationcited by 0
2026ImageAuditor: Membership Inference Attack against Image-based Retrieval-Augmented Generationcited by 0
2026Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrievalcited by 0
2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAGcited by 0
2026DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generationcited by 0
2026When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Modelscited by 0
2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generationcited by 0
2026ViTeGate: Visual-Textual Triggered Knowledge Poisoning for Vision-Language Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way