# Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks

arXiv: https://arxiv.org/abs/2603.00077 · HTML: https://mlsanity.com/paper/2603.00077/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2026.

## Built on by (most cited first, 26 of 26)

- [Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading](https://mlsanity.com/paper/2607.08964/) (2026)
- [SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use](https://mlsanity.com/paper/2607.01874/) (2026)
- [Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why](https://mlsanity.com/paper/2606.00093/) (2026)
- [Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places](https://mlsanity.com/paper/2609.29769/) (2026)
- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)
- [Context-CoT: Enhancing Context Learning via High-Quality Reasoning Synthesis](https://mlsanity.com/paper/2605.25354/) (2026)
- [ComplexConstraints and Beyond: Expert Rubrics for RLVR](https://mlsanity.com/paper/2606.09118/) (2026)
- [A Survey on Rubric-Guided Reinforcement Learning for Language Models](https://mlsanity.com/paper/2608.27505/) (2026)
- [BibTeX Citation Errors in Scientific Publishing Agents: Evaluation and Mitigation](https://mlsanity.com/paper/2604.03159/) (2026)
- [BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows](https://mlsanity.com/paper/2604.11304/) (2026)
- [From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narratives](https://mlsanity.com/paper/2607.00918/) (2026)
- [Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence](https://mlsanity.com/paper/2607.15092/) (2026)
- [Online Monitoring and Corrective Steering of Programming Agents](https://mlsanity.com/paper/2608.06701/) (2026)
- [A Primer in Post-Training Reasoning Data: What We Know About How It Works](https://mlsanity.com/paper/2606.02113/) (2026)
- [Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment](https://mlsanity.com/paper/2606.14948/) (2026)
- [Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs](https://mlsanity.com/paper/2608.01755/) (2026)
- [ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals](https://mlsanity.com/paper/2609.16816/) (2026)
- [Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans](https://mlsanity.com/paper/2610.13303/) (2026)
- [CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging](https://mlsanity.com/paper/2605.19665/) (2026)
- [Using Large Language Models to Support High Volume Application Review for an Undergraduate Research Program](https://mlsanity.com/paper/2606.05564/) (2026)
- [Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents](https://mlsanity.com/paper/2609.33955/) (2026)
- [SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents](https://mlsanity.com/paper/2609.35596/) (2026)
- [The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization](https://mlsanity.com/paper/2609.39055/) (2026)
- [A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined](https://mlsanity.com/paper/2610.01938/) (2026)
- [Evidence-First Agentic Verification](https://mlsanity.com/paper/2610.17724/) (2026)
- [PRISM: Calibrating Black-Box Judges with Counterfactual Probes](https://mlsanity.com/paper/2610.18389/) (2026)

## Abstract (arXiv)

Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks. Yet the underlying judges remain vulnerable to position bias, stochastic inconsistency, criterion conflation, forced judgments under uncertainty, and model-dependent calibration. Rubrics structure evaluation but do not eliminate these failures; they add consequential choices about criterion design, scale types, weighting, aggregation, abstention, calibration, and reliability measurement. Despite extensive research across LLM evaluation, educational measurement, and psychometrics, the relevant methods remain scattered across papers and partial implementations. Researchers therefore pay a reinvention tax, repeatedly rebuilding evaluation machinery instead of accumulating knowledge on a common substrate. We introduce Autorubric, an open-source framework that makes rubric and judge choices explicit, reusable, and auditable. Through a unified API, it supports atomic evaluation of mixed criterion types alongside configurable bias mitigations, calibration, ensembling, abstention, and psychometric diagnostics. Evaluations spanning college chemistry grading, deep-research systems, and CHARM-100---a new mixed-criterion chatbot benchmark---reveal criterion-specific failures, systematic judge-family differences, and configuration effects that do not support a universal mitigation stack. We further demonstrate how per-criterion scores and explanations can support agent skill revision and reward modeling for reinforcement learning. By providing shared infrastructure from measurement through optimization, Autorubric gives the community a common basis for comparing methods, reproducing evaluation choices, and accumulating evidence across studies.