# BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

arXiv: https://arxiv.org/abs/2508.06600 · HTML: https://mlsanity.com/paper/2508.06600/

## Key facts

- Cited by 200 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval](https://mlsanity.com/paper/2407.12883/) (2024)
- [BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese](https://mlsanity.com/paper/2504.19314/) (2025)
- [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://mlsanity.com/paper/2403.20327/) (2024)
- [ReasonIR: Training Retrievers for Reasoning Tasks](https://mlsanity.com/paper/2504.20595/) (2025)
- [An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents](https://mlsanity.com/paper/2505.15117/) (2025)
- [Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality](https://mlsanity.com/paper/2505.02466/) (2025)
- [MedBrowseComp: Benchmarking Medical Deep Research and Computer Use](https://mlsanity.com/paper/2505.14963/) (2025)
- [FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation](https://mlsanity.com/paper/2402.11891/) (2024)
- [DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers](https://mlsanity.com/paper/2502.18460/) (2025)
- [Asyncval: A Toolkit for Asynchronously Validating Dense Retriever Checkpoints during Training](https://mlsanity.com/paper/2202.12510/) (2022)

## Built on by (most cited first, 30 of 200)

- [Scaling Long-Horizon LLM Agent via Context-Folding](https://mlsanity.com/paper/2510.11967/) (2025)
- [Towards a Science of Scaling Agent Systems](https://mlsanity.com/paper/2512.08296/) (2025)
- [Recursive Language Models](https://mlsanity.com/paper/2512.24601/) (2025)
- [WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents](https://mlsanity.com/paper/2509.06501/) (2025)
- [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks](https://mlsanity.com/paper/2602.16313/) (2026)
- [Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2510.12635/) (2025)
- [Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management](https://mlsanity.com/paper/2510.06727/) (2025)
- [SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent](https://mlsanity.com/paper/2511.16108/) (2025)
- [Training Proactive and Personalized LLM Agents](https://mlsanity.com/paper/2511.02208/) (2025)
- [Budget-Aware Tool Use Enables Effective Agent Scaling](https://mlsanity.com/paper/2511.17006/) (2025)
- [OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning](https://mlsanity.com/paper/2603.08655/) (2026)
- [Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction](https://mlsanity.com/paper/2605.05242/) (2026)
- [OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis](https://mlsanity.com/paper/2603.20278/) (2026)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [ClawBench: Can AI Agents Complete Everyday Online Tasks?](https://mlsanity.com/paper/2604.08523/) (2026)
- [MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks](https://mlsanity.com/paper/2601.14652/) (2026)
- [Single-stream Policy Optimization](https://mlsanity.com/paper/2509.13232/) (2025)
- [FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning](https://mlsanity.com/paper/2509.13160/) (2025)
- [Benchmark Test-Time Scaling of General LLM Agents](https://mlsanity.com/paper/2602.18998/) (2026)
- [TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs](https://mlsanity.com/paper/2603.22293/) (2026)
- [Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills](https://mlsanity.com/paper/2512.16301/) (2025)
- [MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents](https://mlsanity.com/paper/2601.12346/) (2026)
- [Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2604.11753/) (2026)
- [Generalizable End-to-End Tool-Use RL with Synthetic CodeGym](https://mlsanity.com/paper/2509.17325/) (2025)
- [BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions](https://mlsanity.com/paper/2510.10666/) (2025)
- [Open Data Synthesis For Deep Research](https://mlsanity.com/paper/2509.00375/) (2025)
- [Towards Personalized Deep Research: Benchmarks and Evaluations](https://mlsanity.com/paper/2509.25106/) (2025)
- [EvoRoute: Experience-Driven Self-Routing LLM Agent Systems](https://mlsanity.com/paper/2601.02695/) (2026)
- [TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents](https://mlsanity.com/paper/2607.13988/) (2026)
- [JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution](https://mlsanity.com/paper/2608.25593/) (2026)

## Abstract (arXiv)

Deep-Research agents, which integrate large language models (LLMs) with search tools, have shown success in improving the effectiveness of handling complex queries that require iterative search planning and reasoning over search results. Evaluations on current benchmarks like BrowseComp relies on black-box live web search APIs, have notable limitations in (1) fairness: dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep research methods; (2) transparency: lack of control over the document corpus makes it difficult to isolate retriever contributions. In other words, the current evaluations may compare a complete deep research system at a given time, but they do not foster well-controlled experiments to provide insights into the capability of underlying deep research LLMs. To address these challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation. The benchmark is shown to be effective in distinguishing the performance of deep research systems. For instance, the open-source model Search-R1, when paired with the BM25 retriever, achieves 3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with fewer search calls. This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods, fostering insights into retrieval effectiveness, citation accuracy, and context engineering in Deep-Research system.