mlsanityData to 7 Oct 2026

RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following

arXiv 2603.25133 ↗First posted 26 Mar 2026cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs). Despite its widespread use, the reliability of these rubric-level evaluations remains unclear, calling for meta-evaluation. However, prior meta-evaluation efforts largely focus on the response level, failing to assess the fine-grained judgment accuracy that rubric-based evaluation relies on. To bridge this gap, we introduce RubricEval. Our benchmark features: (1) the first rubric-level meta-evaluation benchmark for instruction following, (2) diverse instructions and responses spanning multiple categories and model sources, and (3) a substantial set of 3,486 quality-controlled instances, along with Easy/Hard subsets that better differentiates judge performance. Our experiments reveal that rubric-level judging remains far from solved: even GPT-4o, a widely adopted judge in instruction-following benchmarks, achieves only 55.97% on Hard subset. Considering evaluation paradigm, rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and both together reduce inter-judge variance. Through our established rubric taxonomy, we further identify common failure modes and offer actionable insights for reliable instruction-following evaluation.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

22 of 27 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  4. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  5. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  6. 2025Generalizing Verifiable Instruction Followingcited by 178
  7. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  8. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  9. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  10. 2024CFBench: A Comprehensive Constraints-Following Benchmark for LLMscited by 65
  11. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  12. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
10 more
  1. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  2. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  3. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 30
  4. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 28
  5. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  6. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domainscited by 22
  7. 2025UltraIF: Advancing Instruction Following from the Wildcited by 18
  8. 2025Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Modelscited by 12
  9. 2025WildIFEval: Instruction Following in the Wildcited by 10
  10. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 19.

19 papers.

Sort:

By area:

  1. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 20
  2. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  3. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 8
  4. 2026Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?cited by 5
  5. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  6. 2026The Hitchhikers Guide to Rubric Quality Understanding and Enrichmentcited by 3
  7. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  8. 2026Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learningcited by 3
  9. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 2
  10. 2026Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Rankingcited by 2
  11. 2026SEIF: Self-Evolving Reinforcement Learning for Instruction Followingcited by 1
  12. 2026Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubricscited by 1
7 more
  1. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 1
  2. 2026Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototypecited by 0
  3. 2026How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulationcited by 0
  4. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learningcited by 0
  5. 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
  6. 2026PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluatorscited by 0
  7. 2026Evaluating Rubric Generation with Interventional Transfercited by 0