# Self- and Other-Labels Induce Bidirectional Bias in LLM Judges

arXiv: https://arxiv.org/abs/2608.18091 · HTML: https://mlsanity.com/paper/2608.18091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 6 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing](https://mlsanity.com/paper/2507.00769/) (2025)
- [Do LLM Evaluators Prefer Themselves for a Reason?](https://mlsanity.com/paper/2504.03846/) (2025)
- [Beyond the Surface: Measuring Self-Preference in LLM Judgments](https://mlsanity.com/paper/2506.02592/) (2025)
- [The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2509.26072/) (2025)
- [Mitigating Self-Preference by Authorship Obfuscation](https://mlsanity.com/paper/2512.05379/) (2025)
- [Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations](https://mlsanity.com/paper/2601.22548/) (2026)
- [Quantifying Label-Induced Bias in Large Language Model Self- and Cross-Evaluations](https://mlsanity.com/paper/2508.21164/) (2025)
- [Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge](https://mlsanity.com/paper/2604.05593/) (2026)
- [Style over Story: Measuring LLM Narrative Preferences via Structured Selection](https://mlsanity.com/paper/2510.02025/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of "AI-AI Bias" Show No Detectable Own-Model Premium](https://mlsanity.com/paper/2610.00369/) (2026)

## Abstract (arXiv)

As LLM-as-a-judge becomes increasingly widespread, self-preference -- the tendency of a judge to favor its own outputs -- raises growing concerns about evaluation reliability. However, this bias has been studied predominantly on generated text, where stylistic features and response quality are inevitably conflated. As a result, existing measurements cannot separate genuine self-preference from these confounds. We address this limitation by changing the object of evaluation: instead of judging generated text, ten LLMs assess sets of narrative constraints selected from a shared pool, which carry no stylistic fingerprint yet retain a recoverable model-specific signature. Two experiments on this task yield complementary findings. Under blind evaluation, self-preference disappears, with a small effect remaining in the opposite direction once selection quality and judge severity are controlled. Under matched quality, however, self- and other-labels alone -- without naming any model -- shift scores bidirectionally. LLM judges inflate scores for self-labeled selections and deflate those for other-labeled ones regardless of the selection's actual source. We make two contributions: 1) authorship attribution is a distinct driver of evaluation bias, and 2) ground-truth-free tasks can serve as controlled instruments for studying LLM judge behavior.