Language-model groups overstate consensus when replaying human deliberation on a reasoning task
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These complementary routes reduced different measurement asymmetries yet converged within 0.5 percentage points. The gap persisted without early stopping and under a reparameterization removing the memorizable answer; reasoning-mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief-anchored agent groups were biased estimators of the human group-outcome distribution in this setting. These analyses provide a scoring-explicit basis for assessing simulated-group estimates of human deliberative outcomes.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. First posted on arXiv 17 Sep 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AI agents reshape consensus formation in human groups
- 2026From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration
- 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoning
- 2023The Wisdom of Partisan Crowds: Comparing Collective Intelligence in Humans and LLM-based Agents
- 2026GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs
- 2026Toward a social psychology of AI: language-model agents reproduce human-like minimal-group bias
- 2025Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs
- 2026LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systems