MTA-Agent: An Open Recipe for Multimodal Deep Search Agents
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
also in 1 more
Abstract (arXiv)
Multimodal large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning that requires deep searching and integrating visual evidence with external knowledge. In this work, we address this challenge by constructing high-quality, verified multi-hop vision-language training data for multimodal deep-search agents. We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent), which automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources and generates structured multi-hop question-answer trajectories. Starting from diverse VQA seed datasets, our pipeline produces a large-scale training dataset, MTA-Vision-DeepSearch, containing 21K high-quality multi-hop examples. The data is filtered through a multi-stage verification process to ensure factual consistency and answer uniqueness. Using MTA-Vision-DeepSearch, a 32B open-source multimodal search agent achieves state-of-the-art performance, reaching an average of 54.63% across six challenging benchmarks, outperforming GPT-5 (51.86%), Gemini-2.5-Pro (50.98%), and Gemini-3-Pro (54.46%) under the same tool settings. We further show that training on our data improves both reasoning depth and tool-use behavior, increasing the average number of steps from 2.27 to 4.28, and leading to more systematic and persistent search strategies. Additionally, we demonstrate that training can be performed without real-time tool calls by replaying cached interactions, significantly reducing training cost. Importantly, we present MTA-Agent as a fully open recipe for multimodal deep search: we release the entire dataset, training trajectories, and implementation details to enable reproducibility and future research on open multimodal search agents.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 46 references that are in the map. First posted on arXiv 7 Apr 2026.
Builds on
29 of 46 references.
Closest ancestors · 5
Same direction, newest first
- 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models2 Feb
- 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search14 Oct
- 2025MMSearch-R1: Incentivizing LMMs to Search25 Jun
- 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent5 Nov
- 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training31 Jul
Other references · 24
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
- 2025OpenAI GPT-5 System Cardcited by 910
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
- 2021InfographicVQAcited by 430
- 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
- 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
- 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
- 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
- 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
12 more
- 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
- 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 118
- 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
- 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
- 2025A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applicationscited by 48
- 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 35
- 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
- 2025Seeking and Updating with Live Visual Knowledgecited by 23
- 2026MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agentscited by 21
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
- 2026DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agentcited by 8
- 2025Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inferencecited by 6
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agentscited by 3
- 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearchcited by 1
- 2026TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agentscited by 1
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 1
- 2026VaseMuseum: Digital Intelligent Museum for Ancient Greek Potterycited by 0
- 2026Memory-Conditioned Tool Calling for Camera-First Visual Agentscited by 0
- 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrievalcited by 0
- 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval18 Aug 2026
- 2026Memory-Conditioned Tool Calling for Camera-First Visual Agents10 Jul 2026
- 2026VaseMuseum: Digital Intelligent Museum for Ancient Greek Pottery7 Jul 2026
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search30 Jun 2026
- 2026TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents4 Jun 2026
- 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearch2 Jun 2026
- 2026TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents11 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
- 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models
- 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO
- 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents
- 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
- 2026Towards Long-horizon Agentic Multimodal Search
- 2026M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering