Peer Identity Bias in Multi-Agent LLM Evaluation: An Empirical Study Using the TRUST Democratic Discourse Analysis Pipeline
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
The TRUST democratic discourse analysis pipeline exposes its large language model (LLM) components to peer model identity through multiple structural channels -- a design feature whose bias implications have not previously been empirically tested. We provide the first systematic measurement of identity-dependent scoring bias across all active identity exposure channels in TRUST, crossing four model families with two anonymization scopes across 30 political statements. The central finding is that single-channel anonymization produces near-zero bias effects, because individual channels act in opposite directions and cancel each other out -- a result that would lead an evaluator to conclude that identity bias is absent when it is not. Only full-pipeline anonymization reveals the true pattern: homogeneous ensembles amplify identity-driven sycophancy when model identity is fully visible, while the heterogeneous production configuration shows the reverse. Model choice matters independently: one tested model exhibits baseline sycophancy two to three times higher than the others and near-zero deliberative conflict on ideological topics, making it structurally unsuitable for pipelines where genuine inter-role disagreement is the intended quality mechanism. Three practical conclusions follow. First, heterogeneous model ensembles are structurally more robust than homogeneous ones, achieving higher consensus rates and lower identity amplification. Second, full-pipeline anonymization is required for valid bias measurement -- partial anonymization is insufficient and actively misleading. Third, these findings have direct implications for the validation of multi-agent LLM systems in quality-critical applications: a system validated under partial anonymization or with a homogeneous ensemble may pass validation while retaining structural identity bias invisible to single-channel measurement.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 8 of its 17 references that are in the map. First posted on arXiv 24 Apr 2026.
Builds on
8 of 17 references.
Closest ancestors · 2
Same direction, newest first
- 2026From Safety Risk to Design Principle: Peer-Preservation in Multi-Agent LLM Systems and Its Implications for Orchestrated Democratic Discourse Analysis9 Apr
- 2025When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning8 Oct
Other references · 6
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2022Red Teaming Language Models with Language Modelscited by 983
- 2024Alignment faking in large language modelscited by 316
- 2024Do LLMs write like humans? Variation in grammatical and rhetorical stylescited by 62
- 2026Peer-Preservation in Frontier Modelscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis
- 2025An Empirical Study of Group Conformity in Multi-Agent Systems
- 2025Revealing Political Bias in LLMs through Structured Multi-Agent Debate
- 2026LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systems
- 2026How Far Did They Go? The Persuasive Tactics of Covert LLM Agents in a Discontinued Field Experiment
- 2026Beyond Preset Identities: How Agents Form Stances and Boundaries in Generative Societies
- 2025Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rights
- 2025From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactions