mlsanityData to 7 Oct 2026

Improving LLM-as-a-Judge Inference with the Judgment Distribution

arXiv 2503.03064 ↗First posted 4 Mar 2025cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Using language models to scalably approximate human preferences on text quality (LLM-as-a-judge) has become a standard practice applicable to many tasks. A judgment is often extracted from the judge's textual output alone, typically with greedy decoding. However, LLM judges naturally provide distributions over judgment tokens, inviting a breadth of inference methods for extracting fine-grained preferences. We find that taking the mean of the judgment distribution consistently outperforms taking the mode (i.e. greedy decoding) in all evaluation settings (i.e. pointwise, pairwise, and listwise). We further explore novel methods of deriving preferences from judgment distributions, and find that methods incorporating risk aversion often improve performance. Lastly, we analyze LLM-as-a-judge paired with chain-of-thought (CoT) prompting, showing that CoT can collapse the spread of the judgment distribution, often harming performance. Our findings show that leveraging distributional output improves LLM-as-a-judge, as opposed to using the text interface alone.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 57 of its 100 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

57 of 100 references.

References · 57

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
38 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  2. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  4. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  6. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  7. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  8. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  10. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  13. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  14. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  15. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  16. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  17. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  18. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  19. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  20. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  21. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
  22. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  23. 2024FIRST: Faster Improved Listwise Reranking with Single Token Decodingcited by 54
  24. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  25. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  26. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  27. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  28. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  29. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  30. 2024Aligning Crowd Feedback via Distributional Preference Reward Modelingcited by 26
  31. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  32. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  33. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  34. 2024ComPO: Community Preferences for Language Model Personalizationcited by 19
  35. 2024ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedbackcited by 13
  36. 2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
  37. 2024CBEval: A framework for evaluating and interpreting cognitive biases in LLMscited by 12
  38. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 21.

36 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  2. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  3. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  4. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  5. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
  6. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  7. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 10
  8. 2025Measuring Scalar Constructs in Social Science with LLMscited by 8
  9. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  10. 2026ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Modelscited by 6
  11. 2025Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputcited by 5
  12. 2025Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judgecited by 4
24 more
  1. 2025ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"cited by 4
  2. 2025Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Processcited by 4
  3. 2025PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claimscited by 3
  4. 2025ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clusteringcited by 3
  5. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  6. 2026Subliminal Signals in Preference Labelscited by 3
  7. 2026PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournamentscited by 2
  8. 2026When Languages Disagree: Self-Evolving Multilingual LLM Judgescited by 2
  9. 2026When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutationscited by 2
  10. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  11. 2026Permutation-Consensus Listwise Judging for Robust Factuality Evaluationcited by 1
  12. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 1
  13. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1
  14. 2026StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speechcited by 1
  15. 2026Spillover-Aware Multi-Value Steering for Pluralistic LLM Alignmentcited by 1
  16. 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Modelscited by 1
  17. 2025MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixingcited by 0
  18. 2025Approximating Human Preferences Using a Multi-Judge Learned Systemcited by 0
  19. 2026BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluationcited by 0
  20. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  21. 2026BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Taskcited by 0
  22. 2026Can LLMs Rank? A Tale of Triads and Triagecited by 0
  23. 2026Codifying the Judge: Scalable Evaluation via Program Distillationcited by 0
  24. 2026JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocolscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysis
  2. 2026DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
  3. 2025Improve LLM-as-a-Judge Ability as a General Ability
  4. 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes
  5. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
  6. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  7. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  8. 2025Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge