From Web to Pixels: Bringing Agentic Search into Visual Perception
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Visual perception connects high-level semantic understanding to pixel-level perception, but most existing settings assume that the decisive evidence for identifying a target is already in the image or frozen model knowledge. We study a more practical yet harder open-world case where a visible object must first be resolved from external facts, recent events, long-tail entities, or multi-hop relations before it can be localized. We formalize this challenge as Perception Deep Research and introduce WebEye, an object-anchored benchmark with verifiable evidence, knowledge-intensive queries, precise box/mask annotations, and three task views: Search-based Grounding, Search-based Segmentation, and Search-based VQA. WebEyes contains 120 images, 473 annotated object instances, 645 unique QA pairs, and 1,927 task samples. We further propose Pixel-Searcher, an agentic search-to-pixel workflow that resolves hidden target identities and binds them to boxes, masks, or grounded answers. Experiments show that Pixel-Searcher achieves the strongest open-source performance across all three task views, while failures mainly arise from evidence acquisition, identity resolution, and visual instance binding.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 46 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
43 of 46 references.
Closest ancestors · 2
Same direction, newest first
- 2026Gen-Searcher: Reinforcing Agentic Search for Image Generation30 Mar
- 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent7 Aug
Other references · 41
- 2023Segment Anythingcited by 8,234
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
- 2016Modeling Context in Referring Expressionscited by 1,063
- 2025OpenAI GPT-5 System Cardcited by 989
- 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
- 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
29 more
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
- 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
- 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
- 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
- 2016Segmentation from Natural Language Expressionscited by 257
- 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
- 2016FVQA: Fact-based Visual Question Answeringcited by 181
- 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
- 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
- 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
- 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
- 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
- 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
- 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
- 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
- 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 49
- 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
- 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learningcited by 44
- 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
- 2025ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundationscited by 30
- 2025Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Modelcited by 23
- 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
- 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 21
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
- 2026Exploring Reasoning Reward Model for Agentscited by 15
- 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
- 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Searchcited by 6
- 2026ROSE: Retrieval-Oriented Segmentation Enhancementcited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Benchmark Everything Everywhere All at Oncecited by 6
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 2
- 2026EviRover: Reinforcing Agentic Perception Beyond a Glancecited by 0
- 2026EviRover: Reinforcing Agentic Perception Beyond a Glance30 Sep 2026
- 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search30 Jun 2026
- 2026Benchmark Everything Everywhere All at Once4 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seeking
- 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
- 2024Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines
- 2026The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design
- 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding
- 2026Pixelis: Reasoning in Pixels, from Seeing to Acting
- 2026Position: The Systemic Lack of Agency in Visual Reasoning
- 2026Finder: Agentic Closed-Loop Object Finding for Embodied Grounding