mlsanityData to 7 Oct 2026

Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale

arXiv 2601.03444 ↗First posted 6 Jan 2026cs.CL cs.AI cs.HCcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 57 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

33 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges25 Nov
  2. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations5 Aug
  3. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct

Other references · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  12. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
17 more
  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  3. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  4. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  5. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  6. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  7. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  8. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 60
  9. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 44
  10. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  11. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 37
  12. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  13. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  14. 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 23
  15. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?cited by 23
  16. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 21
  17. 2025Evolution without an Oracle: Driving Effective Evolution with LLM Judgescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026MemoBench: Benchmarking World Modeling in Dynamically Changing Environmentscited by 10
  2. 2026LLM Review: Enhancing Creative Writing via Blind Peer Review Feedbackcited by 4
  3. 2026Scaling Small Agents Through Strategy Auctionscited by 3
  4. 2026Vision-Guided Iterative Refinement for Frontend Code Generationcited by 3
  5. 2026Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Modelscited by 1
  6. 2026TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM Jailbreakingcited by 1
  7. 2026EvoTaxo: Building and Evolving Taxonomy from Social Media Streamscited by 1
  8. 2026MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstractscited by 1
  9. 2026Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudgecited by 0
  10. 2026Diagnosing Capability Gaps in Fine-Tuning Datacited by 0
  11. 2026TECCI: Tricky Edits of Collected and Curated Imagescited by 0
  12. 2026MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluatorscited by 0
5 more
  1. 2026Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independencecited by 0
  2. 2026Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Auditcited by 0
  3. 2026The Utility of LLMs in Recommender Systems Explanation Evaluationcited by 0
  4. 2026HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalizationcited by 0
  5. 2026The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
  2. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
  3. 2024Dissecting Human and LLM Preferences
  4. 2026LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora
  5. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  6. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
  7. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
  8. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge