mlsanityData to 14 Oct 2026

Estimating the Error of Large Language Models at Pairwise Text Comparison

arXiv 2510.22219 ↗First posted 25 Oct 2025cs.CL cs.AI math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

We measure LLMs' output error at pairwise text comparison, noting the probability of error in their preferences. Our method does not rely on the ground truth and supports two scenarios: (i) uniform error rate regardless of the order of comparison, estimated with two comparisons for each text pair with either text placed first; (ii) binary positional bias assuming distinct error rates for the two orders of comparison, estimated with repeated comparisons between the texts. The Copeland counting constructs a ranking over the compared texts from pairwise preferences; the ranking reveals the poor scalability of LLM-based pairwise comparison and helps yield the estimates for LLMs' error rates. We apply the method to six LLMs (ChatGPT, Claude, DeepSeek, Gemini, Grok, Qwen) with five types of text input and obtain consistent estimates of LLMs' error. In general, the measured two positional bias terms are similar, close to the uniform error. Considering both the error rates and the robustness to the variation of prompts, Claude obtained the most desirable performance in this experiment. Our model outperforms the biased Bradley-Terry model and the commutativity score in indicating LLMs' error at this task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 25 Oct 2025.

Builds on

23 of 40 references.

References · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  7. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  8. 2023Generating Images with Multimodal Language Modelscited by 260
  9. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  10. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  11. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  12. 2023Deception Abilities Emerged in Large Language Modelscited by 101
11 more
  1. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  2. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 84
  3. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 70
  4. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  5. 2022Efficient computation of rankings from pairwise comparisonscited by 31
  6. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  7. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  8. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  9. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  10. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  11. 2023Testing LLMs on Code Generation with Varying Levels of Prompt Specificitycited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  3. 2026Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
  4. 2026Mutual Debiasing via Dual-Seed Comparison for Probabilistic Sampling in Large Language Models
  5. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  6. 2025Estimating problem difficulty without ground truth using Large Language Model comparisons
  7. 2026A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
  8. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluators