WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search environments often expose retrieved evidence only as text and omit tool-returned images from subsequent context, reducing visually grounded trajectories to text-only reasoning. Long-horizon interaction also compounds tool-call, response-length, timeout, and budget failures, which can discard salvageable trajectories, waste rollout computation, and disturb policy updates. To address these issues, we introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery. Retrieved images receive persistent disk references, allowing the model to inspect, process, and cite them throughout the trajectory. Based on this harness, we develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout. For data construction, a strong MLLM uses WeAgent-Harness to discover, synthesize, and verify MMSearch-style tasks and collect expert trajectories. During post-training, our Failure-Aware GSPO (FA-GSPO) recovers salvageable abnormal rollouts and filters invalid ones to improve bounded multimodal planning and search. We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image, distinguishing image-retrieval failures from visual-perception failures. Evaluation on VisTarget-Bench and seven public benchmarks shows that agentic post-training improves the average score by 19.22 points, enabling our model to outperform similarly sized open-source models and rival models with roughly ten times its parameter count.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 31 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
22 of 31 references.
Closest ancestors · 4
Same direction, newest first
- 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch2 Dec
- 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents29 Aug
- 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents14 Aug
- 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent7 Aug
Other references · 18
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2025Group Sequence Policy Optimizationcited by 788
- 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
- 2016FVQA: Fact-based Visual Question Answeringcited by 181
- 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
- 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
- 2026REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agentscited by 43
- 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
6 more
- 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Modelscited by 33
- 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Modelscited by 30
- 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 18
- 2024Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Enginescited by 14
- 2026Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Modelscited by 13
- 2026HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agentscited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search
- 2026DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing
- 2026ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO
- 2026UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
- 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAG