# Limits of Large Language Models in Debating Humans

arXiv: https://arxiv.org/abs/2402.06049 · HTML: https://mlsanity.com/paper/2402.06049/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Shaping New Norms for AI](https://mlsanity.com/paper/2307.08564/) (2023)

## Built on by (most cited first, 5 of 5)

- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [Y Social: an LLM-powered Social Media Digital Twin](https://mlsanity.com/paper/2408.00818/) (2024)
- [Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research](https://mlsanity.com/paper/2506.01839/) (2025)
- [Computational Turing Test Reveals Systematic Differences Between Human and AI Language](https://mlsanity.com/paper/2511.04195/) (2025)
- [From Perils to Possibilities: Understanding how Human (and AI) Biases affect Online Fora](https://mlsanity.com/paper/2403.14298/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable promise in communicating with humans. Their potential use as artificial partners with humans in sociological experiments involving conversation is an exciting prospect. But how viable is it? Here, we rigorously test the limits of agents that debate using LLMs in a preregistered study that runs multiple debate-based opinion consensus games. Each game starts with six humans, six agents, or three humans and three agents. We found that agents can blend in and concentrate on a debate's topic better than humans, improving the productivity of all players. Yet, humans perceive agents as less convincing and confident than other humans, and several behavioral metrics of humans and agents we collected deviate measurably from each other. We observed that agents are already decent debaters, but their behavior generates a pattern distinctly different from the human-generated data.