# Do as We Do, Not as You Think: the Conformity of Large Language Models

arXiv: https://arxiv.org/abs/2501.13381 · HTML: https://mlsanity.com/paper/2501.13381/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 34 of its 111 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 34)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents](https://mlsanity.com/paper/2405.02957/) (2024)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023)
- [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023)
- [Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents](https://mlsanity.com/paper/2404.16698/) (2024)
- [Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation](https://mlsanity.com/paper/2310.01320/) (2023)
- [Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities](https://mlsanity.com/paper/2407.07791/) (2024)
- [Aligning to Thousands of Preferences via System Message Generalization](https://mlsanity.com/paper/2405.17977/) (2024)
- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM](https://mlsanity.com/paper/2408.12076/) (2024)
- [When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models](https://mlsanity.com/paper/2311.10054/) (2023)
- [Epidemic Modeling with Generative Agents](https://mlsanity.com/paper/2307.04986/) (2023)
- [MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPs](https://mlsanity.com/paper/2408.09955/) (2024)
- [Welfare Diplomacy: Benchmarking Language Model Cooperation](https://mlsanity.com/paper/2310.08901/) (2023)
- [AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation](https://mlsanity.com/paper/2312.13010/) (2023)
- [MEDCO: Medical Education Copilots Based on A Multi-Agent Framework](https://mlsanity.com/paper/2408.12496/) (2024)
- [Multi-Agent Collaboration via Cross-Team Orchestration](https://mlsanity.com/paper/2406.08979/) (2024)
- [Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications](https://mlsanity.com/paper/2311.05876/) (2023)
- [Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System](https://mlsanity.com/paper/2312.04854/) (2023)

## Built on by (most cited first, 30 of 53)

- [Free-MAD: Consensus-Free Multi-Agent Debate](https://mlsanity.com/paper/2509.11035/) (2025)
- [Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2508.05687/) (2025)
- [LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions](https://mlsanity.com/paper/2508.18321/) (2025)
- [Conformity and Social Impact on AI Agents](https://mlsanity.com/paper/2601.05384/) (2026)
- [Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2505.21588/) (2025)
- [The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies](https://mlsanity.com/paper/2509.18052/) (2025)
- [Do we really have to filter out random noise in pre-training data for language models?](https://mlsanity.com/paper/2502.06604/) (2025)
- [Many LLMs Are More Utilitarian Than One](https://mlsanity.com/paper/2507.00814/) (2025)
- [Are Large Language Models Sensitive to the Motives Behind Communication?](https://mlsanity.com/paper/2510.19687/) (2025)
- [Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity](https://mlsanity.com/paper/2606.01637/) (2026)
- [FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems](https://mlsanity.com/paper/2605.11514/) (2026)
- [Disentangling the Drivers of LLM Social Conformity: An Uncertainty-Moderated Dual-Process Mechanism](https://mlsanity.com/paper/2508.14918/) (2025)
- [MADRA: Multi-Agent Debate for Risk-Aware Embodied Planning](https://mlsanity.com/paper/2511.21460/) (2025)
- [Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2601.21742/) (2026)
- [Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systems](https://mlsanity.com/paper/2604.02668/) (2026)
- [LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertainty](https://mlsanity.com/paper/2507.16199/) (2025)
- [Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning](https://mlsanity.com/paper/2512.02914/) (2025)
- [OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering](https://mlsanity.com/paper/2602.03707/) (2026)
- [Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks](https://mlsanity.com/paper/2607.05545/) (2026)
- [Towards Simulating Social Influence Dynamics with LLM-based Multi-agents](https://mlsanity.com/paper/2507.22467/) (2025)
- [Spiral of Silence in Large Language Model Agents](https://mlsanity.com/paper/2510.02360/) (2025)
- [The Effect of Belief Boxes and Open-mindedness on Persuasion](https://mlsanity.com/paper/2512.06573/) (2025)
- [Who Do LLMs Trust? Human Experts Matter More Than Other LLMs](https://mlsanity.com/paper/2602.13568/) (2026)
- [Belief-Calibrated Multi-Agent Consensus Seeking for Complex NLP Tasks](https://mlsanity.com/paper/2510.06307/) (2025)
- [When Agents "Misremember" Collectively: Exploring the Mandela Effect in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2602.00428/) (2026)
- [SinkTrack: Attention Sink based Context Anchoring for Large Language Models](https://mlsanity.com/paper/2604.10027/) (2026)
- [Agentic Microphysics: A Manifesto for Generative AI Safety](https://mlsanity.com/paper/2604.15236/) (2026)
- [The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning](https://mlsanity.com/paper/2605.01704/) (2026)
- [Emergent Collusion in Long-Horizon LLM Agent Interaction](https://mlsanity.com/paper/2609.24967/) (2026)
- [Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable](https://mlsanity.com/paper/2609.37616/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) revolutionize the field of intelligent agents, enabling collaborative multi-agent systems capable of tackling complex problems across various domains. However, the potential of conformity within these systems, analogous to phenomena like conformity bias and groupthink in human group dynamics, remains largely unexplored, raising concerns about their collective problem-solving capabilities and possible ethical implications. This paper presents a comprehensive study on conformity in LLM-driven multi-agent systems, focusing on three aspects: the existence of conformity, the factors influencing conformity, and potential mitigation strategies. In particular, we introduce BenchForm, a new conformity-oriented benchmark, featuring reasoning-intensive tasks and five distinct interaction protocols designed to probe LLMs' behavior in collaborative scenarios. Several representative LLMs are evaluated on BenchForm, using metrics such as conformity rate and independence rate to quantify conformity's impact. Our analysis delves into factors influencing conformity, including interaction time and majority size, and examines how the subject agent rationalizes its conforming behavior. Furthermore, we explore two strategies to mitigate conformity effects, i.e., developing enhanced personas and implementing a reflection mechanism. Several interesting findings regarding LLMs' conformity are derived from empirical results and case studies. We hope that these insights can pave the way for more robust and ethically-aligned collaborative AI systems. Our benchmark and code are available at BenchForm.