# Systematic Biases in LLM Simulations of Debates

arXiv: https://arxiv.org/abs/2402.04049 · HTML: https://mlsanity.com/paper/2402.04049/

## Key facts

- Cited by 99 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Simulating Opinion Dynamics with Networks of LLM-based Agents](https://mlsanity.com/paper/2311.09618/) (2023)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)

## Built on by (most cited first, 30 of 99)

- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey](https://mlsanity.com/paper/2404.11584/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [Determinants of LLM-assisted Decision-Making](https://mlsanity.com/paper/2402.17385/) (2024)
- [Current state of LLM Risks and AI Guardrails](https://mlsanity.com/paper/2406.12934/) (2024)
- [Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks](https://mlsanity.com/paper/2406.17232/) (2024)
- [MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education](https://mlsanity.com/paper/2404.06711/) (2024)
- [100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models](https://mlsanity.com/paper/2505.00551/) (2025)
- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Toward LLM-Agent-Based Modeling of Transportation Systems: A Conceptual Framework](https://mlsanity.com/paper/2412.06681/) (2024)
- [Enhancing Multi-Agent Debate System Performance via Confidence Expression](https://mlsanity.com/paper/2509.14034/) (2025)
- [Unmasking Conversational Bias in AI Multiagent Systems](https://mlsanity.com/paper/2501.14844/) (2025)
- [Does Socialization Emerge in AI Agent Society? A Case Study of Moltbook](https://mlsanity.com/paper/2602.14299/) (2026)
- [GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy](https://mlsanity.com/paper/2407.18008/) (2024)
- [A Survey on Human-Centric LLMs](https://mlsanity.com/paper/2411.14491/) (2024)
- [Explaining Large Language Models Decisions Using Shapley Values](https://mlsanity.com/paper/2404.01332/) (2024)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [A Large-Scale Simulation on Large Language Models for Decision-Making in Political Science](https://mlsanity.com/paper/2412.15291/) (2024)
- [Non-Linear Inference Time Intervention: Improving LLM Truthfulness](https://mlsanity.com/paper/2403.18680/) (2024)
- [Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey](https://mlsanity.com/paper/2503.01513/) (2025)
- [Don't Get Too Excited -- Eliciting Emotions in LLMs](https://mlsanity.com/paper/2503.02457/) (2025)
- [Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity](https://mlsanity.com/paper/2507.00657/) (2025)
- [Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ?](https://mlsanity.com/paper/2410.13517/) (2024)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Evaluating Alignment of Behavioral Dispositions in LLMs](https://mlsanity.com/paper/2602.11328/) (2026)

## Abstract (arXiv)

The emergence of Large Language Models (LLMs), has opened exciting possibilities for constructing computational simulations designed to replicate human behavior accurately. Current research suggests that LLM-based agents become increasingly human-like in their performance, sparking interest in using these AI agents as substitutes for human participants in behavioral studies. However, LLMs are complex statistical learners without straightforward deductive rules, making them prone to unexpected behaviors. Hence, it is crucial to study and pinpoint the key behavioral distinctions between humans and LLM-based agents. In this study, we highlight the limitations of LLMs in simulating human interactions, particularly focusing on LLMs' ability to simulate political debates on topics that are important aspects of people's day-to-day lives and decision-making processes. Our findings indicate a tendency for LLM agents to conform to the model's inherent social biases despite being directed to debate from certain political perspectives. This tendency results in behavioral patterns that seem to deviate from well-established social dynamics among humans. We reinforce these observations using an automatic self-fine-tuning method, which enables us to manipulate the biases within the LLM and demonstrate that agents subsequently align with the altered biases. These results underscore the need for further research to develop methods that help agents overcome these biases, a critical step toward creating more realistic simulations.