mlsanityData to 7 Oct 2026

LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts

arXiv 2501.00274 ↗First posted 31 Dec 2024cs.CLcited by 81

Not on the map. Why?

Abstract (arXiv)

This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges -- indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be textitcombined to textitpredict each human judge's annotations on all questions, including a summary question that assesses overall quality or relevance. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-Rubric with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges' assessment of overall user satisfaction, on a scale of 1--4, with RMS error < 0.5, a 2× improvement over the uncalibrated baseline.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 42 of its 89 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

42 of 89 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2019An Introduction to Variational Autoencoderscited by 997
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
30 more
  1. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  2. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  5. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  6. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  7. 2023Large language models can accurately predict searcher preferencescited by 129
  8. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 96
  9. 2016Neural Autoregressive Distribution Estimationcited by 94
  10. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  11. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  12. 2023Human Feedback is not Gold Standardcited by 82
  13. 2023Large Language Models Enable Few-Shot Clusteringcited by 72
  14. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  15. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 69
  16. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  17. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
  18. 2023Learning to Maximize Mutual Information for Dynamic Feature Selectioncited by 54
  19. 2018Controllable Neural Story Plot Generation via Reward Shapingcited by 49
  20. 2024Multicalibration for Confidence Scoring in LLMscited by 44
  21. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  22. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  23. 2024FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMscited by 27
  24. 2023The Capability of Large Language Models to Measure Psychiatric Functioningcited by 25
  25. 2022Unifying Data Perspectivism and Personalization: An Application to Social Normscited by 22
  26. 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12
  27. 2023Auxiliary Learning as an Asymmetric Bargaining Gamecited by 12
  28. 2024Enhancing Systematic Decompositional Natural Language Inference Using Informal Logiccited by 12
  29. 2020Natural Language Inference with Mixed Effectscited by 9
  30. 2018Dynamic Feature Acquisition Using Denoising Autoencoderscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 60.

81 papers.

Sort:

By area:

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  3. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  4. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  5. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  6. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 42
  7. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  8. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  9. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  10. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  11. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 19
  12. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 19
38 more
  1. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  2. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  3. 2026Step-wise Rubric Rewards for LLM Reasoningcited by 11
  4. 2025Improve LLM-based Automatic Essay Scoring with Linguistic Featurescited by 8
  5. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  6. 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 8
  7. 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
  8. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 8
  9. 2025PentestJudge: Judging Agent Behavior Against Operational Requirementscited by 7
  10. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 7
  11. 2026Many-Tier Instruction Hierarchy in LLM Agentscited by 6
  12. 2025SPRI: Aligning Large Language Models with Context-Situated Principlescited by 5
  13. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  14. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  15. 2025Evaluating Legal Reasoning Traces with Legal Issue Tree Rubricscited by 4
  16. 2026Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revisioncited by 4
  17. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  18. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflowscited by 3
  19. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  20. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 3
  21. 2026The Hitchhikers Guide to Rubric Quality Understanding and Enrichmentcited by 3
  22. 2026Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agentscited by 3
  23. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  24. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  25. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 3
  26. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  27. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  28. 2025Reward Models are Metrics in a Trench Coatcited by 2
  29. 2026Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluationcited by 2
  30. 2026Developing and evaluating a chatbot to support maternal health carecited by 2
  31. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  32. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  33. 2026A Finite-Calibration Regime Map for LLM Judge Panelscited by 2
  34. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  35. 2025RubiSCoT: A Framework for AI-Supported Academic Assessmentcited by 1
  36. 2025AMAP Agentic Planning Technical Reportcited by 1
  37. 2026The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditingcited by 1
  38. 2026Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analyticscited by 1

50 most cited of 81 All 81 (JSON)