DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance. We introduce DRBENCHER, a synthetic benchmark generator for questions that require both browsing and computation, synthesized answer-first from knowledge-graph chains with no seed passages or provenance text. It enforces four criteria: verifiability (gold answers are computed by executing parameterized code over knowledge-graph values), complexity (multi-hop entity identification, property retrieval, and domain-specific computation), difficulty (a two-stage verification cascade filters out questions solvable by the generating model), and diversity (a greedy max-min embedding filter maximizes coverage). These criteria are realized via a unified answer-first pipeline spanning five domains: biochemistry, financial, geophysical, security, and history. Human evaluation shows 76% validity, with 35% of errors due to outdated knowledge-graph entries, highlighting an inherent limitation of systems that reason over evolving data. Automatic evaluation shows that the strongest frontier model achieves only 20% answer accuracy. Compared to manually constructed benchmarks (BrowseComp+, MATH-500, GPQA), DRBENCHER achieves the highest semantic diversity. We release DrBencher pipeline and the human-verified benchmark data at https://github.com/IBM/DrBencher.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 32 of its 50 references that are in the map. First posted on arXiv 10 Apr 2026.
Builds on
32 of 50 references.
Closest ancestors · 3
Same direction, newest first
- 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling14 Nov
- 2025WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models16 Oct
- 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training1 Aug
Other references · 29
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
17 more
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
- 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
- 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
- 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
- 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
- 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
- 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 118
- 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
- 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 76
- 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
- 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 40
- 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
- 2025Granite Embedding R2 Modelscited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports
- 2026AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
- 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
- 2025Improving and Evaluating Open Deep Research Agents
- 2026DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
- 2026From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents
- 2026DR-Arena: an Automated Evaluation Framework for Deep Research Agents
- 2026DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent