mlsanityData to 14 Oct 2026

Quantifying and Mitigating Self-Preference Bias of LLM Judges

arXiv 2604.22891 ↗First posted 24 Apr 2026cs.LG cs.AI cs.CLcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on. However, the scalability and trustworthiness of this approach can be substantially distorted by Self-Preference Bias (SPB), which is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation. Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems. To address this issue, we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards. Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB. To mitigate this bias, we propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5% on average.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 63 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

33 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Do LLM Evaluators Prefer Themselves for a Reason?4 Apr
  2. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  10. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  12. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
18 more
  1. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  3. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  4. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  5. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  6. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  7. 2024Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden Statescited by 52
  8. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  9. 2025Kimi-Dev: Agentless Training as Skill Prior for SWE-Agentscited by 41
  10. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  11. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  12. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
  13. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 30
  14. 2023NBIAS: A Natural Language Processing Framework for Bias Identification in Textcited by 23
  15. 2025Information Suppression in Large Language Models: Auditing, Quantifying, and Characterizing Censorship in DeepSeekcited by 11
  16. 2024Can I understand what I create? Self-Knowledge Evaluation of Large Language Modelscited by 6
  17. 2024Evaluating the quality of published medical research with ChatGPTcited by 3
  18. 2025Extreme Self-Preference in Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  2. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 8
  3. 2026LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliabilitycited by 6
  4. 2026Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skillcited by 3
  5. 2026Can LLM Coding Agents Reason About Time Series?cited by 1
  6. 2026Mitigating Fabrication in Multi-Stage LLM Pipelines for Hiring: An Empirical Evaluation of Prompt Guardrails and Human-in-the-Loop Checkpointscited by 1
  7. 2026Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panelscited by 1
  8. 2026Automated IEP Generation from Traditional Chinese Parent-Teacher Interviews via Corpus-Grounded Feature Diffusioncited by 0
  9. 2026A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of "AI-AI Bias" Show No Detectable Own-Model Premiumcited by 0
  10. 2026A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
  2. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  3. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  4. 2025Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
  5. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
  6. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  7. 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
  8. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysis