mlsanityData to 7 Oct 2026

Judging with Confidence: Calibrating Autoraters to Preference Distributions

arXiv 2510.00263 ↗First posted 30 Sep 2025cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 68 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

42 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 40

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
28 more
  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  2. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  3. 2023Instruction Tuning with GPT-4cited by 676
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  10. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  11. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  12. 2023Nash Learning from Human Feedbackcited by 220
  13. 2023Generative Judge for Evaluating Alignmentcited by 163
  14. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  15. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  16. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  17. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  18. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  19. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  20. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  21. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  22. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  23. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
  24. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  25. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  26. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  27. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 29
  28. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Linear Social Choice with Few Queries: A Moment-Based Approachcited by 1
  2. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1
  3. 2026Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling
  2. 2026Response Time Enhances Alignment with Heterogeneous Preferences
  3. 2024Aligning language models with human preferences
  4. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts