# Evaluating the Consistency of LLM Evaluators

arXiv: https://arxiv.org/abs/2412.00543 · HTML: https://mlsanity.com/paper/2412.00543/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 30 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)

## Built on by (most cited first, 24 of 24)

- [Improving LLM-as-a-Judge Inference with the Judgment Distribution](https://mlsanity.com/paper/2503.03064/) (2025)
- [A Technical Survey of Reinforcement Learning Techniques for Large Language Models](https://mlsanity.com/paper/2507.04136/) (2025)
- [Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale](https://mlsanity.com/paper/2601.03444/) (2026)
- [Media and responsible AI governance: a game-theoretic and LLM analysis](https://mlsanity.com/paper/2503.09858/) (2025)
- [Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents](https://mlsanity.com/paper/2504.08640/) (2025)
- [Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning](https://mlsanity.com/paper/2503.07002/) (2025)
- [CAPE: Context-Aware Personality Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2508.20385/) (2025)
- [The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks](https://mlsanity.com/paper/2509.09705/) (2025)
- [DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses](https://mlsanity.com/paper/2512.02282/) (2025)
- [HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions](https://mlsanity.com/paper/2505.15087/) (2025)
- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://mlsanity.com/paper/2507.18442/) (2025)
- [A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretability](https://mlsanity.com/paper/2502.12052/) (2025)
- [CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation](https://mlsanity.com/paper/2509.07325/) (2025)
- [CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations](https://mlsanity.com/paper/2512.23711/) (2025)
- [What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025](https://mlsanity.com/paper/2601.07648/) (2026)
- [Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge](https://mlsanity.com/paper/2605.16023/) (2026)
- [Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment](https://mlsanity.com/paper/2605.19529/) (2026)
- [Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis](https://mlsanity.com/paper/2505.17636/) (2025)
- [How Good LLM-Generated Password Policies Are?](https://mlsanity.com/paper/2506.08320/) (2025)
- [A Comprehensive Evaluation framework of Alignment Techniques for LLMs](https://mlsanity.com/paper/2508.09937/) (2025)
- [Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation](https://mlsanity.com/paper/2510.14915/) (2025)
- [MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following](https://mlsanity.com/paper/2605.03858/) (2026)
- [Agent Step Value: Auditing Evaluation-Channel Sensitivity in Black-Box Agent Traces](https://mlsanity.com/paper/2607.04419/) (2026)
- [Query Timing Produces Opposite Positional Biases Between LLMs and Humans](https://mlsanity.com/paper/2608.12387/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown potential as general evaluators along with the evident benefits of speed and cost. While their correlation against human annotators has been widely studied, consistency as evaluators is still understudied, raising concerns about the reliability of LLM evaluators. In this paper, we conduct extensive studies on the two aspects of consistency in LLM evaluations, Self-Consistency (SC) and Inter-scale Consistency (IC), on different scoring scales and criterion granularity with open-source and proprietary models. Our comprehensive analysis demonstrates that strong proprietary models are not necessarily consistent evaluators, highlighting the importance of considering consistency in assessing the capability of LLM evaluators.