mlsanityData to 7 Oct 2026

Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

arXiv 2408.13006 ↗First posted 23 Aug 2024cs.CLcited by 71

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

LLM-as-a-Judge has been widely applied to evaluate and compare different LLM alignmnet approaches (e.g., RLHF and DPO). However, concerns regarding its reliability have emerged, due to LLM judges' biases and inconsistent decision-making. Previous research has developed evaluation frameworks to assess reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address LLM internal inconsistency. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-Judge methods, leading to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM-as-a-Judge on alignment tasks by defining more theoretically interpretable evaluation metrics and explicitly mitigating LLM internal inconsistency from reliability metrics. We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges, which facilitates practitioners to choose LLM judges for alignment tasks. In the experiments, we examine effects of diverse prompt templates on LLM-judge reliability and also demonstrate our developed framework by comparing various LLM judges on two common alignment datasets (i.e., TL;DR Summarization and HH-RLHF-Helpfulness). Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 61 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

48 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  4. 2023Training Language Models with Language Feedback at Scale28 Mar
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
31 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  4. 2024Self-Rewarding Language Modelscited by 654
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  7. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  10. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  13. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  14. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  15. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  16. 2023Generative Judge for Evaluating Alignmentcited by 163
  17. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  18. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  19. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  20. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  21. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  22. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  23. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  24. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
  25. 2024Variational Best-of-N Alignmentcited by 49
  26. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  27. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  28. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
  29. 2024ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Modelscited by 15
  30. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsalcited by 14
  31. 2024Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 42; 2026: 21.

71 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  3. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  4. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  5. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  6. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  7. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  8. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  9. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 39
  10. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  11. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  12. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
38 more
  1. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 21
  2. 2025AI Alignment at Your Discretioncited by 20
  3. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  4. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17
  5. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 14
  6. 2024DHP Benchmark: Are LLMs Good NLG Evaluators?cited by 12
  7. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  8. 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
  9. 2025PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?cited by 9
  10. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 9
  11. 2025EnronQA: Towards Personalized RAG over Private Documentscited by 8
  12. 2025INTIMA: A Benchmark for Human-AI Companionship Behaviorcited by 8
  13. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 7
  14. 2025Benchmarking Multi-National Value Alignment for Large Language Modelscited by 7
  15. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  16. 2026PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Supportcited by 7
  17. 2025AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasonerscited by 6
  18. 2025Quantitative LLM Judgescited by 6
  19. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 6
  20. 2025Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systemscited by 6
  21. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
  22. 2025Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluationcited by 5
  23. 2024Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluatorcited by 4
  24. 2025F`ux`i: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generationcited by 4
  25. 2025AI Propaganda factories with language modelscited by 4
  26. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
  27. 2026AI Organizations are More Effective but Less Aligned than Individual Agentscited by 4
  28. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 4
  29. 2025LLMs Judging LLMs: A Simplex Perspectivecited by 3
  30. 2025Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judgescited by 3
  31. 2025Understanding the Dilemma of Unlearning for Large Language Modelscited by 3
  32. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 3
  33. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  34. 2026The Necessity of Setting Temperature in LLM-as-a-Judgecited by 3
  35. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Biascited by 3
  36. 2024UCFE: A User-Centric Financial Expertise Benchmark for Large Language Modelscited by 2
  37. 2025Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Realitycited by 2
  38. 2025Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluationcited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  2. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  3. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
  4. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
  5. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference
  6. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
  7. 2025The Differences Between Direct Alignment Algorithms are a Blur
  8. 2025Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge