mlsanityData to 7 Oct 2026

Bayesian Calibration of Win Rate Estimation with LLM Evaluators

arXiv 2411.04424 ↗First posted 7 Nov 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 24 of its 51 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

24 of 51 references.

References · 24

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2015Teaching Machines to Read and Comprehendcited by 1,039
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
12 more
  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  3. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  4. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  5. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  6. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  7. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  8. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  9. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  10. 2014Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcingcited by 63
  11. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  12. 2024LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extractioncited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 7.

15 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  5. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8
  6. 2026From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judgescited by 6
  7. 2026STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systemscited by 3
  8. 2024The Superalignment of Superhuman Intelligence with Large Language Modelscited by 2
  9. 2025LLMPopcorn: Exploring LLMs as Assistants for Popular Micro-video Generationcited by 2
  10. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  11. 2026PETS: A Principled Framework Towards Optimal Trajectory Allocation for Efficient Test-Time Self-Consistencycited by 2
  12. 2025Sequential Bayesian Evaluation of Large Language Model Behaviorcited by 1
3 more
  1. 2026Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluationcited by 1
  2. 2026BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validitycited by 0
  3. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0