mlsanityData to 14 Oct 2026

Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge

arXiv 2505.15240 ↗First posted 21 May 2025cs.AI cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper explores generalised probabilistic modelling and uncertainty estimation in comparative LLM-as-a-judge frameworks. We show that existing Product-of-Experts methods are specific cases of a broader framework, enabling diverse modelling options. Furthermore, we propose improved uncertainty estimates for individual comparisons, enabling more efficient selection and achieving strong performance with fewer evaluations. We also introduce a method for estimating overall ranking uncertainty. Finally, we demonstrate that combining absolute and comparative scoring improves performance. Experiments show that the specific expert model has a limited impact on final rankings but our proposed uncertainty estimates, especially the probability of reordering, significantly improve the efficiency of systems reducing the number of needed comparisons by ~50%. Furthermore, ranking-level uncertainty metrics can be used to identify low-performing predictions, where the nature of the probabilistic model has a notable impact on the quality of the overall uncertainty.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

29 of 47 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
17 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  2. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  5. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  6. 2023GPTScore: Evaluate as You Desirecited by 361
  7. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  8. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  9. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  10. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 75
  11. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  12. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  13. 2022Efficient computation of rankings from pairwise comparisonscited by 31
  14. 2023Primacy Effect of ChatGPTcited by 27
  15. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  16. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  17. 2024Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truthcited by 7
  2. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  3. 2026CIExplainer++: Generating Causal and Interpretable Explanations for Graph Neural Networkscited by 0
  4. 2026OSCAR: Order-aware Sensitivity Correction for AI Rankingscited by 0