# Source framing triggers systematic evaluation bias in Large Language Models

arXiv: https://arxiv.org/abs/2505.13488 · HTML: https://mlsanity.com/paper/2505.13488/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2025.

## Built on by (most cited first, 3 of 3)

- [How Similar Are Grokipedia and Wikipedia? A Multi-Dimensional Textual and Structural Comparison](https://mlsanity.com/paper/2510.26899/) (2025)
- [Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physics](https://mlsanity.com/paper/2605.10528/) (2026)
- [Domain Adaptation and Reasoning Frameworks in Language Models: A Controlled Experiment with Historical Cosmology](https://mlsanity.com/paper/2605.30415/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used not only to generate text but also to evaluate it, raising urgent questions about whether their judgments are consistent, unbiased, and robust to framing effects. In this study, we systematically examine inter- and intra-model agreement across four state-of-the-art LLMs (OpenAI o3-mini, Deepseek Reasoner, xAI Grok 2, and Mistral) tasked with evaluating 4,800 narrative statements on 24 different topics of social, political, and public health relevance, for a total of 192,000 assessments. We manipulate the disclosed source of each statement to assess how attribution to either another LLM or a human author of specified nationality affects evaluation outcomes. We find that, in the blind condition, different LLMs display a remarkably high degree of inter- and intra-model agreement across topics. However, this alignment breaks down when source framing is introduced. Here we show that attributing statements to Chinese individuals systematically lowers agreement scores across all models, and in particular for Deepseek Reasoner. Our findings reveal that framing effects can deeply affect text evaluation, with significant implications for the integrity, neutrality, and fairness of LLM-mediated information systems.