OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents
Not on the map. Why?
Abstract (arXiv)
Deep search has become a crucial capability for frontier multimodal agents, enabling models to solve complex questions through active search, evidence verification, and multi-step reasoning. Despite rapid progress, top-tier multimodal search agents remain difficult to reproduce, largely due to the absence of open high-quality training data, transparent trajectory synthesis pipelines, or detailed training recipes. To this end, we introduce OpenSearch-VL, a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning. First, we curated a dedicated pipeline to construct high-quality training data through Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding, which jointly reduce shortcuts and one-step retrieval collapse. Based on this pipeline, we curate two training datasets, SearchVL-SFT-36k for SFT and SearchVL-RL-8k for RL. Besides, we design a diverse tool environment that unifies text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition. Finally, we propose a multi-turn fatal-aware GRPO training algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping. Built on this recipe, OpenSearch-VL delivers substantial performance gains, with over 10-point average improvements across seven benchmarks, and achieves results comparable to proprietary commercial models on several tasks. We will release all data, code, and models to support open research on multimodal deep search agents.
Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 6 May 2026.
Builds on
41 of 49 references.
References · 41
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2025OpenAI GPT-5 System Cardcited by 989
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
29 more
- 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
- 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
- 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
- 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
- 2016FVQA: Fact-based Visual Question Answeringcited by 181
- 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
- 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
- 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 119
- 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 66
- 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
- 2025Visual Agentic Reinforcement Fine-Tuningcited by 55
- 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 53
- 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 42
- 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Modelscited by 37
- 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Modelscited by 34
- 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
- 2026Gen-Searcher: Reinforcing Agentic Search for Image Generationcited by 30
- 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 27
- 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 22
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
- 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 17
- 2026Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesiscited by 17
- 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
- 2026Exploring Reasoning Reward Model for Agentscited by 15
- 2025Perception in Reflectioncited by 13
- 2026AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoningcited by 7
- 2026OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Taskscited by 2
- 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalizationcited by 1
- 2026Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generationcited by 1
Built on by
Citing papers per year: 2026: 21.
21 papers.
Sort:
By area:
- 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agentscited by 8
- 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoningcited by 8
- 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
- 2026SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulationcited by 5
- 2026From Web to Pixels: Bringing Agentic Search into Visual Perceptioncited by 3
- 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearchcited by 3
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 2
- 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agentscited by 2
- 2026Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agentcited by 2
- 2026VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?cited by 2
- 2026WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agentscited by 1
- 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understandingcited by 0
9 more
- 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrievalcited by 0
- 2026Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agentscited by 0
- 2026AdaVDR: Adaptive Tool Use and Reflection for Video Deep Researchcited by 0
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPOcited by 0
- 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratificationcited by 0
- 2026Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agentscited by 0
- 2026OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Videocited by 0
- 2026Neos: Learning to Interleave Search and Visual Reasoningcited by 0
- 2026ActICL: Post-Training for Agentic Multimodal In-Context Learningcited by 0
- 2026ActICL: Post-Training for Agentic Multimodal In-Context Learning14 Oct 2026
- 2026Neos: Learning to Interleave Search and Visual Reasoning12 Oct 2026
- 2026OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video8 Oct 2026
- 2026Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents4 Oct 2026
- 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification7 Sep 2026
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO29 Aug 2026
- 2026WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents28 Aug 2026
- 2026AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research26 Aug 2026
- 2026Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents24 Aug 2026
- 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval18 Aug 2026
- 2026VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?15 Aug 2026
- 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding12 Aug 2026
9 more
- 2026Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent4 Aug 2026
- 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents3 Aug 2026
- 2026SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation7 Jul 2026
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search30 Jun 2026
- 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning13 Jun 2026
- 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment8 Jun 2026
- 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearch2 Jun 2026
- 2026From Web to Pixels: Bringing Agentic Search into Visual Perception12 May 2026
- 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents11 May 2026
- 2026From Web to Pixels: Bringing Agentic Search into Visual Perceptioncited by 3
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 2
- 2026Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agentcited by 2
- 2026VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?cited by 2
- 2026WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agentscited by 1
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPOcited by 0
- 2026Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agentscited by 0
- 2026OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Videocited by 0
- 2026OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video8 Oct 2026
- 2026Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents4 Oct 2026
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO29 Aug 2026
- 2026WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents28 Aug 2026
- 2026VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?15 Aug 2026
- 2026Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent4 Aug 2026
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search30 Jun 2026
- 2026From Web to Pixels: Bringing Agentic Search into Visual Perception12 May 2026
- 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agentscited by 8
- 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoningcited by 8
- 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
- 2026SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulationcited by 5
- 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratificationcited by 0
- 2026Neos: Learning to Interleave Search and Visual Reasoningcited by 0
- 2026Neos: Learning to Interleave Search and Visual Reasoning12 Oct 2026
- 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification7 Sep 2026
- 2026SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation7 Jul 2026
- 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning13 Jun 2026
- 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment8 Jun 2026
- 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents11 May 2026
- 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearchcited by 3
- 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agentscited by 2
- 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understandingcited by 0
- 2026ActICL: Post-Training for Agentic Multimodal In-Context Learningcited by 0
- 2026ActICL: Post-Training for Agentic Multimodal In-Context Learning14 Oct 2026
- 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding12 Aug 2026
- 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents3 Aug 2026
- 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearch2 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
- 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning
- 2026Towards Long-horizon Agentic Multimodal Search
- 2026OpenSeeker-v2: Pushing the Limits of Search Agents with Informative and High-Difficulty Trajectories
- 2026POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management
- 2026InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search
- 2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agents
- 2026OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data