# Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

arXiv: https://arxiv.org/abs/2506.21506 · HTML: https://mlsanity.com/paper/2506.21506/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 47 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 14 of 14)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025)
- [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [MindSearch: Mimicking Human Minds Elicits Deep AI Searcher](https://mlsanity.com/paper/2407.20183/) (2024)
- [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks](https://mlsanity.com/paper/2401.13649/) (2024)
- [BEARCUBS: A benchmark for computer-using web agents](https://mlsanity.com/paper/2503.07919/) (2025)

## Built on by (most cited first, 30 of 76)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers](https://mlsanity.com/paper/2508.14704/) (2025)
- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks](https://mlsanity.com/paper/2602.16313/) (2026)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry](https://mlsanity.com/paper/2507.16280/) (2025)
- [DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle](https://mlsanity.com/paper/2512.04324/) (2025)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [Test-Time Adaptation for LLM Agents via Environment Interaction](https://mlsanity.com/paper/2511.04847/) (2025)
- [WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks](https://mlsanity.com/paper/2601.02439/) (2026)
- [TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications](https://mlsanity.com/paper/2510.18586/) (2025)
- [LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?](https://mlsanity.com/paper/2509.21501/) (2025)
- [ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning](https://mlsanity.com/paper/2509.25586/) (2025)
- [DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation](https://mlsanity.com/paper/2512.17776/) (2025)
- [ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration](https://mlsanity.com/paper/2509.21823/) (2025)
- [How Far Are We from Genuinely Useful Deep Research Agents?](https://mlsanity.com/paper/2512.01948/) (2025)
- [QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks](https://mlsanity.com/paper/2605.24218/) (2026)
- [LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services](https://mlsanity.com/paper/2512.07436/) (2025)
- [RecGPT-V2 Technical Report](https://mlsanity.com/paper/2512.14503/) (2025)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation](https://mlsanity.com/paper/2510.19205/) (2025)
- [AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping](https://mlsanity.com/paper/2602.12315/) (2026)
- [TimeWarp: Evaluating Web Agents by Revisiting the Past](https://mlsanity.com/paper/2603.04949/) (2026)
- [Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents](https://mlsanity.com/paper/2507.05495/) (2025)
- [DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories](https://mlsanity.com/paper/2602.10809/) (2026)
- [Atomicity for Agents: Exposing, Exploiting, and Mitigating TOCTOU Vulnerabilities in Browser-Use Agents](https://mlsanity.com/paper/2603.00476/) (2026)
- [When Is Enough Not Enough? Illusory Completion in Search Agents](https://mlsanity.com/paper/2602.07549/) (2026)
- [Sequential Behavioral Watermarking for LLM Agents](https://mlsanity.com/paper/2605.11036/) (2026)
- [SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills](https://mlsanity.com/paper/2605.24117/) (2026)
- [Watermarking LLM Agent Trajectories](https://mlsanity.com/paper/2602.18700/) (2026)

## Abstract (arXiv)

Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.