mlsanityData to 7 Oct 2026

To Mask or to Mirror: Human-AI Alignment in Collective Reasoning

arXiv 2510.01924 ↗First posted 2 Oct 2025cs.AI cs.MAcited by 5

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

As large language models (LLMs) are increasingly used to model and augment collective decision-making, it is critical to examine their alignment with human social reasoning. We present an empirical framework for assessing collective alignment, in contrast to prior work on the individual level. Using the Lost at Sea social psychology task, we conduct a large-scale online experiment (N=748), randomly assigning groups to leader elections with either visible demographic attributes (e.g. name, gender) or pseudonymous aliases. We then simulate matched LLM groups conditioned on the human data, benchmarking Gemini 2.5, GPT 4.1, Claude Haiku 3.5, and Gemma 3. LLM behaviors diverge: some mirror human biases; others mask these biases and attempt to compensate for them. We empirically demonstrate that human-AI alignment in collective reasoning depends on context, cues, and model-specific inductive biases. Understanding how LLMs align with collective human behavior is critical to advancing socially-aligned AI, and demands dynamic benchmarks that capture the complexities of collective reasoning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 67 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

27 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models1 Nov

Other references · 26

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  3. 2021Ethical and social risks of harm from Language Modelscited by 957
  4. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  5. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 235
  6. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 228
  7. 2023Simulating Opinion Dynamics with Networks of LLM-based Agentscited by 153
  8. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  9. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  10. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  11. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  12. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
14 more
  1. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
  2. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 61
  3. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  4. 2023Do LLM Agents Exhibit Social Behavior?cited by 53
  5. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 33
  6. 2023The Wisdom of Partisan Crowds: Comparing Collective Intelligence in Humans and LLM-based Agentscited by 30
  7. 2025Language Agents as Digital Representatives in Collective Decision-Makingcited by 15
  8. 2024The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Makingcited by 12
  9. 2025Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargainingcited by 11
  10. 2024Investigating LLMs as Voting Assistants via Contextual Augmentation: A Case Study on the European Parliament Elections 2024cited by 6
  11. 2024A Study of Nationality Bias in Names and Perplexity using Off-the-Shelf Affect-related Tweet Classifierscited by 4
  12. 2023Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?cited by 3
  13. 2024Evaluating LLMs for Gender Disparities in Notable Personscited by 3
  14. 2022Assessing Group-level Gender Bias in Professional Evaluations: The Case of Medical Student End-of-Shift Feedbackcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Does Socialization Emerge in AI Agent Society? A Case Study of Moltbookcited by 13
  2. 2026Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humanscited by 4
  3. 2026Discovering Differences in Strategic Behavior Between Humans and LLMscited by 2
  4. 2026Memory as Metabolism: A Design for Companion Knowledge Systemscited by 0
  5. 2026Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Normscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Right and Wrong: Evaluating Second-order Social Reasoning in Large Language Models
  2. 2024LLM Voting: Human Choices and AI Collective Decision Making
  3. 2026Toward a social psychology of AI: language-model agents reproduce human-like minimal-group bias
  4. 2025Computational Basis of LLM's Decision Making in Social Simulation
  5. 2026Social Reasoning in Machines: Investigating Collective Truth-Seeking Dynamics in Large Language Model Debate
  6. 2025Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
  7. 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysis
  8. 2024Investigating social alignment via mirroring in a system of interacting language models