mlsanityData to 7 Oct 2026

ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

arXiv 2511.07685 ↗First posted 10 Nov 2025cs.AI cs.CL cs.LGcited by 80

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics(including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 26 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

20 of 26 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks1 Sep
  2. 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks14 Aug
  3. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge26 Jun
  4. 2025Deep Research Agents: A Systematic Examination And Roadmap22 Jun
  5. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents13 Jun
  6. 2025DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research25 May

Other references · 14

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  4. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  5. 2025Humanity's Last Examcited by 467
  6. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  7. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  8. 2025ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decompositioncited by 57
  9. 2025A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applicationscited by 48
  10. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  11. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 31
  12. 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 29
2 more
  1. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  2. 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 75.

80 papers.

Sort:

By area:

  1. 2026Kimi K3: Open Frontier Intelligencecited by 99
  2. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  3. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  4. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 22
  5. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  6. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  7. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  8. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  9. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  10. 2025Step-DeepResearch Technical Reportcited by 12
  11. 2026DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivitycited by 11
  12. 2026DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factualitycited by 10
38 more
  1. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 10
  2. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  3. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 9
  4. 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Modelcited by 9
  5. 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 8
  6. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 8
  7. 2026QUEST: Training Frontier Deep Research Agents with Fully Synthetic Taskscited by 8
  8. 2026From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agentscited by 7
  9. 2026DREAM: Deep Research Evaluation with Agentic Metricscited by 7
  10. 2026Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generationcited by 7
  11. 2026The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Researchcited by 6
  12. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
  13. 2026Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproductioncited by 6
  14. 2026Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agentscited by 5
  15. 2026Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?cited by 5
  16. 2026Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revisioncited by 4
  17. 2026Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarkscited by 4
  18. 2026RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinementcited by 4
  19. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 4
  20. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  21. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  22. 2026DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agentscited by 4
  23. 2026Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomiescited by 3
  24. 2026LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Taskscited by 3
  25. 2026The Hitchhikers Guide to Rubric Quality Understanding and Enrichmentcited by 3
  26. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 3
  27. 2026Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learningcited by 3
  28. 2026S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agentscited by 3
  29. 2026FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agentscited by 3
  30. 2026Self-Optimizing Multi-Agent Systems for Deep Researchcited by 2
  31. 2026Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systemscited by 2
  32. 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
  33. 2026K-Bench: measuring model performance on real scientific agent requestscited by 2
  34. 2026Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluationcited by 2
  35. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  36. 2026Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articlescited by 1
  37. 2026Hunt Globally: Wide Search AI Agents for Drug Asset Scouting in Investing, Business Development, and Competitive Intelligencecited by 1
  38. 2026PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncologycited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports
  2. 2026DR-Arena: an Automated Evaluation Framework for Deep Research Agents
  3. 2025Deep Research: A Survey of Autonomous Research Agents
  4. 2025ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents
  5. 2025Improving and Evaluating Open Deep Research Agents
  6. 2026From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents
  7. 2025ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry
  8. 2026Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Science