mlsanityData to 7 Oct 2026

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

arXiv 2410.02736 ↗First posted 3 Oct 2024cs.CL cs.AIcited by 354

Not on the map. Why?

Cited by 40 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework-CALM-which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.

Key facts. Cited by 354 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

44 of 61 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
32 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  7. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  8. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  9. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  10. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  11. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  12. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  13. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  14. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  15. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  16. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  17. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  18. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  19. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  20. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  21. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 80
  22. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  23. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  24. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  25. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  26. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understandingcited by 42
  27. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  28. 2024Evaluating Large Language Models with Psychometricscited by 22
  29. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  30. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 20
  31. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16
  32. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settingscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 154; 2026: 188.

354 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  5. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  6. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  7. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  8. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  9. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  10. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  11. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  12. 2025One Token to Fool LLM-as-a-Judgecited by 56
38 more
  1. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  2. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
  3. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  4. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  5. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  6. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 39
  7. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  8. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  9. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  10. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  11. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  12. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 32
  13. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  14. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  15. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
  16. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  17. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  18. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  19. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  20. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  21. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  22. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  23. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  24. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  25. 2025DyFlow: Dynamic Workflow Framework for Agentic Reasoningcited by 22
  26. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 22
  27. 2025When Large Language Models are Reliable for Judging Empathic Communicationcited by 21
  28. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  29. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
  30. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  31. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  32. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  33. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17
  34. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  35. 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16
  36. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 16
  37. 2025REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformationscited by 15
  38. 2025OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientistscited by 15

50 most cited of 354 All 354 (JSON)