# Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale arXiv: https://arxiv.org/abs/2601.03444 ยท HTML: https://mlsanity.com/paper/2601.03444/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. - Builds on 33 of its 57 references that are in the map. - First posted on arXiv 6 Jan 2026. ## Placed in - LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/) ## Builds on (most cited first, 30 of 33) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024) - [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023) - [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017) - [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023) - [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024) - [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020) - [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023) - [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024) - [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023) - [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024) - [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023) - [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023) - [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023) - [GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation](https://mlsanity.com/paper/2401.04092/) (2024) - [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024) - [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018) - [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://mlsanity.com/paper/2505.23802/) (2025) - [FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model](https://mlsanity.com/paper/2406.06004/) (2024) - [Evaluating Scoring Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2506.22316/) (2025) - [RocketEval: Efficient Automated LLM Evaluation via Grading Checklist](https://mlsanity.com/paper/2503.05142/) (2025) - [Investigating Non-Transitivity in LLM-as-a-Judge](https://mlsanity.com/paper/2502.14074/) (2025) - [Validating LLM-as-a-Judge Systems under Rating Indeterminacy](https://mlsanity.com/paper/2503.05965/) (2025) - [Evaluating the Consistency of LLM Evaluators](https://mlsanity.com/paper/2412.00543/) (2024) - [Are We on the Right Way to Assessing LLM-as-a-Judge?](https://mlsanity.com/paper/2512.16041/) (2025) - [INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback](https://mlsanity.com/paper/2305.14282/) (2023) ## Built on by (most cited first, 20 of 20) - [MemoBench: Benchmarking World Modeling in Dynamically Changing Environments](https://mlsanity.com/paper/2606.27537/) (2026) - [LLM Review: Enhancing Creative Writing via Blind Peer Review Feedback](https://mlsanity.com/paper/2601.08003/) (2026) - [Scaling Small Agents Through Strategy Auctions](https://mlsanity.com/paper/2602.02751/) (2026) - [Vision-Guided Iterative Refinement for Frontend Code Generation](https://mlsanity.com/paper/2604.05839/) (2026) - [Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models](https://mlsanity.com/paper/2601.07984/) (2026) - [TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM Jailbreaking](https://mlsanity.com/paper/2602.06440/) (2026) - [EvoTaxo: Building and Evolving Taxonomy from Social Media Streams](https://mlsanity.com/paper/2603.19711/) (2026) - [MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts](https://mlsanity.com/paper/2604.06505/) (2026) - [Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence](https://mlsanity.com/paper/2608.25869/) (2026) - [The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators](https://mlsanity.com/paper/2609.21133/) (2026) - [Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge](https://mlsanity.com/paper/2604.03742/) (2026) - [Diagnosing Capability Gaps in Fine-Tuning Data](https://mlsanity.com/paper/2604.27547/) (2026) - [TECCI: Tricky Edits of Collected and Curated Images](https://mlsanity.com/paper/2606.01213/) (2026) - [MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators](https://mlsanity.com/paper/2608.18096/) (2026) - [Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit](https://mlsanity.com/paper/2608.27309/) (2026) - [The Utility of LLMs in Recommender Systems Explanation Evaluation](https://mlsanity.com/paper/2609.01627/) (2026) - [HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization](https://mlsanity.com/paper/2609.09835/) (2026) - [How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models](https://mlsanity.com/paper/2610.05094/) (2026) - [JudgeMoE: Distributional Aggregation for LLM-as-a-Judge](https://mlsanity.com/paper/2610.07109/) (2026) - [The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express](https://mlsanity.com/paper/2610.17158/) (2026) ## Abstract (arXiv) Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.