mlsanityData to 7 Oct 2026

Language-model groups overstate consensus when replaying human deliberation on a reasoning task

arXiv 2609.20543 ↗First posted 17 Sep 2026cs.AI cs.CL cs.CYcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These complementary routes reduced different measurement asymmetries yet converged within 0.5 percentage points. The gap persisted without early stopping and under a reparameterization removing the memorizable answer; reasoning-mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief-anchored agent groups were biased estimators of the human group-outcome distribution in this setting. These analyses provide a scoring-explicit basis for assessing simulated-group estimates of human deliberative outcomes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. First posted on arXiv 17 Sep 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AI agents reshape consensus formation in human groups
  2. 2026From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration
  3. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoning
  4. 2023The Wisdom of Partisan Crowds: Comparing Collective Intelligence in Humans and LLM-based Agents
  5. 2026GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs
  6. 2026Toward a social psychology of AI: language-model agents reproduce human-like minimal-group bias
  7. 2025Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs
  8. 2026LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systems