mlsanityData to 7 Oct 2026

MMSearch-R1: Incentivizing LMMs to Search

arXiv 2506.20670 ↗First posted 25 Jun 2025cs.CV cs.CLcited by 120

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Robust deployment of large multimodal models (LMMs) in real-world scenarios requires access to external knowledge sources, given the complexity and dynamic nature of real-world information. Existing approaches such as retrieval-augmented generation (RAG) and prompt engineered search agents rely on rigid pipelines, often leading to inefficient or excessive search behaviors. We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables LMMs to perform on-demand, multi-turn search in real-world Internet environments. Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty. To support training, We collect a multimodal search VQA dataset through a semi-automated pipeline that covers diverse visual and textual knowledge needs and curate a search-balanced subset with both search-required and search-free samples, which proves essential for shaping efficient and on-demand search behavior. Extensive experiments on knowledge-intensive and info-seeking VQA tasks show that our model not only outperforms RAG-based baselines of the same model size, but also matches the performance of a larger RAG-based model while reducing search calls by over 30%. We further analyze key empirical findings to offer actionable insights for advancing research in multimodal search.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 73 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

55 of 73 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines28 Oct
  2. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training31 Jul
  3. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models13 May
  4. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
38 more
  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  13. 2016A Diagram Is Worth A Dozen Imagescited by 793
  14. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  15. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  16. 2025Kimi-VL Technical Reportcited by 381
  17. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  18. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  19. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  20. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  21. 2023Demystifying CLIP Datacited by 265
  22. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  23. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  24. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  25. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  26. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  27. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  28. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  29. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  30. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  31. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  32. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
  33. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  34. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  35. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 47
  36. 2025Open Deep Search: Democratizing Search with Open-source Reasoning Agentscited by 46
  37. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  38. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 37; 2026: 83.

120 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  3. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  4. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  5. 2025Deep Research: A Survey of Autonomous Research Agentscited by 63
  6. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  7. 2026REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agentscited by 43
  8. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  9. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  10. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agentscited by 39
  11. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  12. 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Modelscited by 33
38 more
  1. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
  2. 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Modelscited by 30
  3. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  4. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  5. 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 25
  6. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 24
  7. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  8. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  9. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  10. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  11. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
  12. 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 18
  13. 2025Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learningcited by 17
  14. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 16
  15. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  16. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  17. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  18. 2025FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysiscited by 13
  19. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  20. 2026Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Modelscited by 13
  21. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  22. 2026Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaborationcited by 10
  23. 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Spacecited by 10
  24. 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
  25. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  26. 2026VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agentscited by 8
  27. 2026HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agentscited by 8
  28. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  29. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answeringcited by 7
  30. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 7
  31. 2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agentscited by 7
  32. 2025RepoForge: Training a SOTA Fast-thinking SWE Agent with an End-to-End Data Curation Pipeline Synergizing SFT and RL at Scalecited by 6
  33. 2025LLM-I: LLMs are Naturally Interleaved Multimodal Creatorscited by 6
  34. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  35. 2025Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safetycited by 6
  36. 2026VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videoscited by 6
  37. 2026SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glassescited by 6
  38. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learningcited by 6

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback
  2. 2025Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning
  3. 2025M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation
  4. 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents
  5. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
  6. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searching
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
  8. 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents