mlsanityData to 14 Oct 2026

BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

arXiv 2508.06600 ↗First posted 8 Aug 2025cs.CL cs.IRcited by 200

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Deep-Research agents, which integrate large language models (LLMs) with search tools, have shown success in improving the effectiveness of handling complex queries that require iterative search planning and reasoning over search results. Evaluations on current benchmarks like BrowseComp relies on black-box live web search APIs, have notable limitations in (1) fairness: dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep research methods; (2) transparency: lack of control over the document corpus makes it difficult to isolate retriever contributions. In other words, the current evaluations may compare a complete deep research system at a given time, but they do not foster well-controlled experiments to provide insights into the capability of underlying deep research LLMs. To address these challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation. The benchmark is shown to be effective in distinguishing the performance of deep research systems. For instance, the open-source model Search-R1, when paired with the BM25 retriever, achieves 3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with fewer search calls. This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods, fostering insights into retrieval effectiveness, citation accuracy, and context engineering in Deep-Research system.

Key facts. Cited by 200 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 38 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

27 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agent3 Jul
  2. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents21 May

Other references · 25

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  11. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  12. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
13 more
  1. 2023Improving Text Embeddings with Large Language Modelscited by 303
  2. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  3. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  4. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  5. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  6. 2025BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinesecited by 145
  7. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 83
  8. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 77
  9. 2025Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modalitycited by 30
  10. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 30
  11. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  12. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  13. 2022Asyncval: A Toolkit for Asynchronously Validating Dense Retriever Checkpoints during Trainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 165.

200 papers.

Sort:

By area:

  1. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 148
  2. 2025Towards a Science of Scaling Agent Systemscited by 138
  3. 2025Recursive Language Modelscited by 104
  4. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 92
  5. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  6. 2025Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Taskscited by 66
  7. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  8. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 48
  9. 2025Training Proactive and Personalized LLM Agentscited by 46
  10. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 44
  11. 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoningcited by 39
  12. 2026Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interactioncited by 38
38 more
  1. 2026OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesiscited by 37
  2. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  3. 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 33
  4. 2026MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarkscited by 32
  5. 2025Single-stream Policy Optimizationcited by 28
  6. 2025FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoningcited by 27
  7. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 26
  8. 2026TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMscited by 24
  9. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  10. 2026MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agentscited by 22
  11. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  12. 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 17
  13. 2025BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actionscited by 17
  14. 2025Open Data Synthesis For Deep Researchcited by 16
  15. 2025Towards Personalized Deep Research: Benchmarks and Evaluationscited by 16
  16. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 16
  17. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  18. 2026JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolutioncited by 16
  19. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  20. 2026Learning Agent-Compatible Context Management for Long-Horizon Taskscited by 15
  21. 2026DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluationcited by 14
  22. 2026LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growthcited by 14
  23. 2026ContextBudget: Budget-Aware Context Management for Long-Horizon Search Agentscited by 14
  24. 2026MemoBrain: Executive Memory as an Agentic Brain for Reasoningcited by 13
  25. 2026General Agent Evaluationcited by 13
  26. 2026AgentIR: Reasoning-Aware Retrieval for Deep Research Agentscited by 13
  27. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  28. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  29. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  30. 2026Coding Agents are Effective Long-Context Processorscited by 12
  31. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
  32. 2025How Far Are We from Genuinely Useful Deep Research Agents?cited by 11
  33. 2026SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoningcited by 11
  34. 2026Towards Long-horizon Agentic Multimodal Searchcited by 11
  35. 2026AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discoverycited by 11
  36. 2026QUEST: Training Frontier Deep Research Agents with Fully Synthetic Taskscited by 11
  37. 2026Beyond Semantic Organization: Memory as Execution State Management for Long-Horizon Agentscited by 11
  38. 2026Self-Compacting Language Model Agentscited by 11

50 most cited of 200 All 200 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving and Evaluating Open Deep Research Agents
  2. 2026BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents
  3. 2026Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search
  4. 2025Deep Research Bench: Evaluating AI Web Research Agents
  5. 2025DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research
  6. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
  7. 2025ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework
  8. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents