# Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement

arXiv: https://arxiv.org/abs/2402.11436 · HTML: https://mlsanity.com/paper/2402.11436/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Can Large Language Models Really Improve by Self-critiquing Their Own Plans?](https://mlsanity.com/paper/2310.08118/) (2023)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems](https://mlsanity.com/paper/2310.12397/) (2023)
- [LLMs cannot find reasoning errors, but can correct them given the error location](https://mlsanity.com/paper/2311.08516/) (2023)
- [CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation](https://mlsanity.com/paper/2311.18702/) (2023)
- [A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2311.07954/) (2023)

## Built on by (most cited first, 30 of 114)

- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024)
- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models](https://mlsanity.com/paper/2411.17451/) (2024)
- [Grounded Chain-of-Thought for Multimodal Large Language Models](https://mlsanity.com/paper/2503.12799/) (2025)
- [The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input](https://mlsanity.com/paper/2501.03200/) (2025)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Internal Consistency and Self-Feedback in Large Language Models: A Survey](https://mlsanity.com/paper/2407.14507/) (2024)
- [Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2505.20272/) (2025)
- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [Do LLM Evaluators Prefer Themselves for a Reason?](https://mlsanity.com/paper/2504.03846/) (2025)
- [Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2508.06709/) (2025)
- [Beyond the Surface: Measuring Self-Preference in LLM Judgments](https://mlsanity.com/paper/2506.02592/) (2025)
- [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://mlsanity.com/paper/2505.18761/) (2025)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [Safety Aware Task Planning via Large Language Models in Robotics](https://mlsanity.com/paper/2503.15707/) (2025)
- [From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks](https://mlsanity.com/paper/2409.04168/) (2024)
- [Brittlebench: Quantifying LLM robustness via prompt sensitivity](https://mlsanity.com/paper/2603.13285/) (2026)
- [Learning Code Preference via Synthetic Evolution](https://mlsanity.com/paper/2410.03837/) (2024)
- [Variation in Verification: Understanding Verification Dynamics in Large Language Models](https://mlsanity.com/paper/2509.17995/) (2025)
- [Can Automatic Metrics Assess High-Quality Translations?](https://mlsanity.com/paper/2405.18348/) (2024)
- [Improving Model Factuality with Fine-grained Critique-based Evaluator](https://mlsanity.com/paper/2410.18359/) (2024)
- [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025)
- [Generative AI Act II: Test Time Scaling Drives Cognition Engineering](https://mlsanity.com/paper/2504.13828/) (2025)
- [Depression Diagnosis Dialogue Simulation: Self-improving Psychiatrist with Tertiary Memory](https://mlsanity.com/paper/2409.15084/) (2024)
- [MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2410.13716/) (2024)
- [Multi-expert Prompting Improves Reliability, Safety, and Usefulness of Large Language Models](https://mlsanity.com/paper/2411.00492/) (2024)

## Abstract (arXiv)

Recent studies show that large language models (LLMs) improve their performance through self-feedback on certain tasks while degrade on others. We discovered that such a contrary is due to LLM's bias in evaluating their own output. In this paper, we formally define LLM's self-bias - the tendency to favor its own generation - using two statistics. We analyze six LLMs (GPT-4, GPT-3.5, Gemini, LLaMA2, Mixtral and DeepSeek) on translation, constrained text generation, and mathematical reasoning tasks. We find that self-bias is prevalent in all examined LLMs across multiple languages and tasks. Our analysis reveals that while the self-refine pipeline improves the fluency and understandability of model outputs, it further amplifies self-bias. To mitigate such biases, we discover that larger model size and external feedback with accurate assessment can significantly reduce bias in the self-refine pipeline, leading to actual performance improvement in downstream tasks. The code and data are released at https://github.com/xu1998hz/llm_self_bias.