mlsanityData to 7 Oct 2026

The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators

arXiv 2406.12319 ↗First posted 18 Jun 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) are increasingly used as evaluators for natural language generation tasks, ensuring unbiased assessments is essential. However, LLM evaluators often display biased preferences, such as favoring verbosity and authoritative tones. Our empirical analysis reveals that these biases are exacerbated in pairwise evaluation, where LLMs directly compare two outputs and easily prioritize superficial attributes. In contrast, pointwise evaluation, which assesses outputs independently, is less susceptible to such bias because each output is judged in isolation. To address the limitations of the pairwise evaluation, we introduce a novel evaluation method, PRePair, which integrates pointwise reasoning within a pairwise framework. PRePair effectively alleviates biased preference, improving performance on the adversarial benchmark (LLMBar) while outperforming pointwise evaluation on the standard benchmark (MT-Bench).

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

25 of 43 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
13 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  6. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  7. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  8. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  9. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  10. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  11. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  12. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  13. 2023PromptBench: A Unified Library for Evaluation of Large Language Modelscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8
  2. 2025Quantitative LLM Judgescited by 6
  3. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
  4. 2026Understanding LLM Evaluator Behavior: A Structured Multi-Evaluator Framework for Merchant Risk Assessmentcited by 1
  5. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  6. 2025CLINB: A Climate Intelligence Benchmark for Foundational Modelscited by 0
  7. 2026PRISM: Learning Design Knowledge from Data for Stylistic Design Improvementcited by 0
  8. 2026Active Learners as Efficient PRP Rerankerscited by 0
  9. 2026Side-by-side Comparison Amplifies Dialect Bias in Language Modelscited by 0
  10. 2026Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Studycited by 0
  11. 2026Backbone-Adaptive Evidence Routing for Robust Pairwise LLM Judgingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
  2. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
  3. 2025Estimating the Error of Large Language Models at Pairwise Text Comparison
  4. 2024Self-Preference Bias in LLM-as-a-Judge
  5. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
  6. 2026Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
  7. 2025Do LLM Evaluators Prefer Themselves for a Reason?
  8. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluators