mlsanityData to 14 Oct 2026

Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

arXiv 2506.21506 ↗First posted 26 Jun 2025cs.AI cs.CLcited by 76

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 47 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

14 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025An Illusion of Progress? Assessing the Current State of Web Agents2 Apr
  2. 2025BEARCUBS: A benchmark for computer-using web agents10 Mar
  3. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?22 Jul
  4. 2024WebCanvas: Benchmarking Web Agents in Online Environments18 Jun
  5. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks24 Jan

Other references · 9

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2024OpenAI o1 System Cardcited by 1,244
  4. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  5. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  6. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  7. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  8. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  9. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 66

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 55.

76 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  3. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 84
  4. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  5. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  6. 2025ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquirycited by 32
  7. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 23
  8. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  9. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 20
  10. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  11. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  12. 2025LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?cited by 15
38 more
  1. 2025ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planningcited by 13
  2. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  3. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 11
  4. 2025How Far Are We from Genuinely Useful Deep Research Agents?cited by 11
  5. 2026QUEST: Training Frontier Deep Research Agents with Fully Synthetic Taskscited by 11
  6. 2025LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Servicescited by 8
  7. 2025RecGPT-V2 Technical Reportcited by 8
  8. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  9. 2025WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representationcited by 7
  10. 2026AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shoppingcited by 7
  11. 2026TimeWarp: Evaluating Web Agents by Revisiting the Pastcited by 7
  12. 2025Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agentscited by 6
  13. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  14. 2026Atomicity for Agents: Exposing, Exploiting, and Mitigating TOCTOU Vulnerabilities in Browser-Use Agentscited by 6
  15. 2026When Is Enough Not Enough? Illusory Completion in Search Agentscited by 5
  16. 2026Sequential Behavioral Watermarking for LLM Agentscited by 5
  17. 2026SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skillscited by 5
  18. 2026Watermarking LLM Agent Trajectoriescited by 4
  19. 2026SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?cited by 4
  20. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 4
  21. 2025Look It Up: Analysing Internal Web Search Capabilities of Modern LLMscited by 3
  22. 2026ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Researchcited by 3
  23. 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Modelscited by 3
  24. 2026Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planningcited by 3
  25. 2026Online Skill Learning for Web Agents via State-Grounded Dynamic Retrievalcited by 3
  26. 2025SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Modelscited by 2
  27. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Modelscited by 2
  28. 2026The Verifiable Action Card: Trustworthy Human-in-the-Loop Control for Secure Autonomous Agentscited by 2
  29. 2025WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Taskscited by 1
  30. 2025Improving and Evaluating Open Deep Research Agentscited by 1
  31. 2026AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systemscited by 1
  32. 2026Persuasion Propagation: Does Persuasion Change AI Agent Behavior?cited by 1
  33. 2026From Human-Level AI Tales to AI Leveling Human Scalescited by 1
  34. 2026DV-World: Benchmarking Data Visualization Agents in Real-World Scenarioscited by 1
  35. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  36. 2026ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reductioncited by 1
  37. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agentscited by 1
  38. 2026DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivationcited by 1

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
  2. 2026DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks
  3. 2025Deep Research Bench: Evaluating AI Web Research Agents
  4. 2025RAVine: Reality-Aligned Evaluation for Agentic Search
  5. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
  6. 2026AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web
  7. 2026Web2BigTable: A Bi-Level Multi-Agent LLM System for Internet-Scale Information Search and Extraction
  8. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents