mlsanityData to 7 Oct 2026

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

arXiv 2605.17554 ↗First posted 17 May 2026cs.AI cs.LGcited by 1

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Frontier deep research agents (DRAs) are being deployed in enterprise workflows faster than they are being evaluated. Existing benchmarks measure factual recall, single-hop QA, or generic agentic skill, and miss the multi-document, decision-grade deliverables DRAs are asked to produce. We introduce a benchmark of 70 SME-authored management consulting prompts, each embedding cognitive traps that penalize surface-pattern reasoning. Three frontier agents, namely Claude Opus~4.6, OpenAI o3-deep-research and Gemini~3.1~Pro deep-research, are scored on two complementary layers: deterministic binary verifiers (mean 14.9 per task) and a five-criterion 0--3 SME rubric (Data Integrity, Analytical Rigor, Relevance & Focus, Execution Precision, Format & Deliverability), combined into a Verifier-Rubric Score (VRS, 0--100). Acceptance under a joint threshold (rubric mean ≥ 2.5 and verifier pass rate ≥ 80%) is uniformly low: o3 15.7%, Claude 12.9%, Gemini 12.9%. Pairwise differences are statistically indistinguishable. On the continuous VRS, o3 leads (61.4~[CI: 55.2,,67.5]), followed by Gemini (52.6) and Claude (38.5); the o3--Claude gap (Delta=22.9, p<0.001) survives Bonferroni correction. No agent averages above the rubric's ``adequate'' threshold of 2.0; no agent's mean verifier pass rate reaches the 80% acceptance floor. Each agent fails distinctively: Claude leads on data fabrication and file-access failures; o3 propagates cascading computation errors; Gemini oscillates between the highest perfect-verifier rate and the most catastrophic collapses. The benchmark, evaluation code, and full prompt corpus are publicly released.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 50 of its 75 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

50 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents10 Nov
  2. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents13 Jun
  3. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoning18 Feb
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul

Other references · 46

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
34 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  13. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  14. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  15. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  16. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  17. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  18. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  19. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  20. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  21. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  22. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  23. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  24. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  25. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  26. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  27. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  28. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  29. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  30. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  31. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
  32. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  33. 2025ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquirycited by 30
  34. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
  2. 2025How Far Are We from Genuinely Useful Deep Research Agents?
  3. 2025Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports
  4. 2026Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback
  5. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification
  6. 2025ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents
  7. 2025Deep Research Agents: A Systematic Examination And Roadmap
  8. 2025Deep Research Bench: Evaluating AI Web Research Agents