mlsanityData to 14 Oct 2026

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge

arXiv 2406.07791 ↗First posted 12 Jun 2024cs.CL cs.AIcited by 269

Not on the map. Why?

Cited by 51 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLM-as-a-Judge has emerged as a promising alternative to human evaluators across various tasks, yet inherent biases - particularly position bias, the tendency to favor solutions based on their position within the prompt - compromise its reliability. This exploratory study evaluates position bias in LLM judges across pairwise and list-wise comparison settings, introducing three metrics: repetition stability, position consistency, and preference fairness. Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances. We identify Judge-Level, Candidate-Level, and Task-Level factors contributing to bias. The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions. Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.

Key facts. Cited by 269 papers in the map (data through 14 Oct 2026). Builds on 21 of its 39 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

21 of 39 references.

References · 21

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  7. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  8. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  9. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  10. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  11. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
9 more
  1. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  2. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  3. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  4. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  5. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  6. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  7. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
  8. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thoughtcited by 39
  9. 2023Large Language Models are Miscalibrated In-Context Learnerscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 49; 2026: 216.

269 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  3. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  4. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 45
  5. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 43
  6. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 31
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  8. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judgescited by 24
  9. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 22
  10. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 19
  11. 2025Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanismcited by 16
  12. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
38 more
  1. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 15
  2. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  3. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 14
  4. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
  5. 2025AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognitioncited by 13
  6. 2025Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processingcited by 12
  7. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  8. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automationcited by 10
  9. 2025Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Modelscited by 10
  10. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 10
  11. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  12. 2026The Illusion of Insight in Reasoning Modelscited by 9
  13. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  14. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  15. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  16. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  17. 2026Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Modelscited by 8
  18. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 8
  19. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 8
  20. 2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
  21. 2025SSR: Socratic Self-Refine for Large Language Model Reasoningcited by 7
  22. 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluationcited by 7
  23. 2026Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generationcited by 7
  24. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  25. 2026LOGIGEN: Logic-Driven Generation of Verifiable Agentic Taskscited by 7
  26. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systemscited by 7
  27. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 6
  28. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  29. 2026Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judgecited by 6
  30. 2026SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behaviorcited by 6
  31. 2026LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliabilitycited by 6
  32. 2026GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluationcited by 6
  33. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 6
  34. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  35. 2025MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answeringcited by 5
  36. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 5
  37. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judgecited by 5
  38. 2026Characterizing, Evaluating, and Optimizing Complex Reasoningcited by 5

50 most cited of 269 All 269 (JSON)