# Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents arXiv: https://arxiv.org/abs/2608.31076 ยท HTML: https://mlsanity.com/paper/2608.31076/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. - Builds on 28 of its 43 references that are in the map. - First posted on arXiv 31 Aug 2026. ## Placed in - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 28 of 28) - [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024) - [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026) - [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023) - [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024) - [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025) - [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025) - [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025) - [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024) - [Kosmos: An AI Scientist for Autonomous Discovery](https://mlsanity.com/paper/2511.02824/) (2025) - [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025) - [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025) - [Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks](https://mlsanity.com/paper/2602.05125/) (2026) - [Autonomous LLM-driven research from data to human-verifiable research papers](https://mlsanity.com/paper/2404.17605/) (2024) - [AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite](https://mlsanity.com/paper/2510.21652/) (2025) - [DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports](https://mlsanity.com/paper/2601.08536/) (2026) - [Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification](https://mlsanity.com/paper/2601.15808/) (2026) - [ResearchGym: Evaluating Language Model Agents on Real-World AI Research](https://mlsanity.com/paper/2602.15112/) (2026) - [SedarEval: Automated Evaluation using Self-Adaptive Rubrics](https://mlsanity.com/paper/2501.15595/) (2025) - [Learning to Judge: LLMs Designing and Applying Evaluation Rubrics](https://mlsanity.com/paper/2602.08672/) (2026) - [StructMem: Structured Memory for Long-Horizon Behavior in LLMs](https://mlsanity.com/paper/2604.21748/) (2026) - [AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation](https://mlsanity.com/paper/2605.28655/) (2026) - [Qworld: Question-Specific Evaluation Criteria for LLMs](https://mlsanity.com/paper/2603.23522/) (2026) - [Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge](https://mlsanity.com/paper/2605.30568/) (2026) - [Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction](https://mlsanity.com/paper/2607.12835/) (2026) - [Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance](https://mlsanity.com/paper/2605.07461/) (2026) - [RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement](https://mlsanity.com/paper/2605.09730/) (2026) - [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026) - [LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation](https://mlsanity.com/paper/2608.12990/) (2026) ## Abstract (arXiv) Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).