mlsanityData to 7 Oct 2026

Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

arXiv 2510.22694 ↗First posted 26 Oct 2025cs.CV cs.CL cs.IRcited by 1

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

51 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering25 Nov
  2. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA22 Nov
  3. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information21 Sep
  4. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training31 Jul
  5. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul
  6. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs29 May
  7. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr
  8. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
31 more
  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  5. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  6. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  7. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  8. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  9. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  10. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  11. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  12. 2021WebQA: Multihop and Multimodal QAcited by 109
  13. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  14. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  15. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  16. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  17. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  18. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  19. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  20. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  21. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  22. 2024Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuningcited by 27
  23. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 24
  24. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  25. 2023MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugincited by 22
  26. 2022QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillationcited by 21
  27. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  28. 2023NEUCORE: Neural Concept Reasoning for Composed Image Retrievalcited by 8
  29. 2025Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learningcited by 5
  30. 2021Rescuing Deep Hashing from Dead Bits Problemcited by 3
  31. 2024KALAHash: Knowledge-Anchored Low-Resource Adaptation for Deep Hashingcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025MoRA: Missing Modality Low-Rank Adaptation for Visual Recognitioncited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  2. 2025A Survey of Multimodal Retrieval-Augmented Generation
  3. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
  4. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts
  5. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning
  6. 2025CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation
  7. 2025Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation
  8. 2025M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation