# Unmasking Conversational Bias in AI Multiagent Systems

arXiv: https://arxiv.org/abs/2501.14844 · HTML: https://mlsanity.com/paper/2501.14844/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 81 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 30 of 44)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Gender bias and stereotypes in Large Language Models](https://mlsanity.com/paper/2308.14921/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [Simulating Opinion Dynamics with Networks of LLM-based Agents](https://mlsanity.com/paper/2311.09618/) (2023)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [The Political Preferences of LLMs](https://mlsanity.com/paper/2402.01789/) (2024)
- [How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://mlsanity.com/paper/2212.14548/) (2022)
- [Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs](https://mlsanity.com/paper/2310.07251/) (2023)
- [Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?](https://mlsanity.com/paper/2406.13605/) (2024)
- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [MoralBench: Moral Evaluation of LLMs](https://mlsanity.com/paper/2406.04428/) (2024)
- [The Persuasive Power of Large Language Models](https://mlsanity.com/paper/2312.15523/) (2023)

## Built on by (most cited first, 17 of 17)

- [The simulation of judgment in LLMs](https://mlsanity.com/paper/2502.04426/) (2025)
- [Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamics](https://mlsanity.com/paper/2506.12657/) (2025)
- [Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity](https://mlsanity.com/paper/2507.00657/) (2025)
- [A Tutorial on Cognitive Biases in Agentic AI-Driven 6G Autonomous Networks](https://mlsanity.com/paper/2510.19973/) (2025)
- [Toward an Unbiased Collective Memory for Efficient LLM-Based Agentic 6G Cross-Domain Management](https://mlsanity.com/paper/2509.26200/) (2025)
- [LLM-Based Agentic Negotiation for 6G: Addressing Uncertainty Neglect and Tail-Event Risk](https://mlsanity.com/paper/2511.19175/) (2025)
- [MOSAIC: Unveiling the Moral, Social and Individual Dimensions of Large Language Models](https://mlsanity.com/paper/2603.00048/) (2026)
- [What if AI systems weren't chatbots?](https://mlsanity.com/paper/2605.07896/) (2026)
- [From Personalization to Prejudice: Bias and Discrimination in Memory-Enhanced AI Agents for Recruitment](https://mlsanity.com/paper/2512.16532/) (2025)
- [Harm in AI-Driven Societies: An Audit of Toxicity Adoption on Chirper.ai](https://mlsanity.com/paper/2601.01090/) (2026)
- [AI Biases as Asymmetries: A Review to Guide Practice](https://mlsanity.com/paper/2503.07326/) (2025)
- [Strategic Reflectivism In Intelligent Systems](https://mlsanity.com/paper/2505.22987/) (2025)
- [Exploring Implicit Perspectives on Autism in Large Language Models Through Multi-Agent Simulations](https://mlsanity.com/paper/2601.15437/) (2026)
- [Fairness in Multi-Agent Systems for Software Engineering: An SDLC-Oriented Rapid Review](https://mlsanity.com/paper/2604.13103/) (2026)
- [Mitigating Anchoring Bias in LLM-Based Agents for Energy-Efficient 6G Autonomous Networks](https://mlsanity.com/paper/2606.18272/) (2026)
- [Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study](https://mlsanity.com/paper/2608.11649/) (2026)
- [Peer Influence across Heterogeneous AI Models](https://mlsanity.com/paper/2610.03095/) (2026)

## Abstract (arXiv)

Detecting biases in the outputs produced by generative models is essential to reduce the potential risks associated with their application in critical settings. However, the majority of existing methodologies for identifying biases in generated text consider the models in isolation and neglect their contextual applications. Specifically, the biases that may arise in multi-agent systems involving generative models remain under-researched. To address this gap, we present a framework designed to quantify biases within multi-agent systems of conversational Large Language Models (LLMs). Our approach involves simulating small echo chambers, where pairs of LLMs, initialized with aligned perspectives on a polarizing topic, engage in discussions. Contrary to expectations, we observe significant shifts in the stance expressed in the generated messages, particularly within echo chambers where all agents initially express conservative viewpoints, in line with the well-documented political bias of many LLMs toward liberal positions. Crucially, the bias observed in the echo-chamber experiment remains undetected by current state-of-the-art bias detection methods that rely on questionnaires. This highlights a critical need for the development of a more sophisticated toolkit for bias detection and mitigation for AI multi-agent systems. The code to perform the experiments is publicly available.