mlsanityData to 14 Oct 2026

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation

arXiv 2605.01986 ↗First posted 3 May 2026cs.AIcited by 1

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

What if the twelve jurors of Sidney Lumet's 12 Angry Men (1957) were not men, but large language models? Would the one juror who disagrees still be able to change everyone's mind? This paper instantiates that scenario as a multi-agent benchmark for LLM deliberation: twelve agents, each conditioned on a film-faithful persona, debate the film's murder case using multi-agent framework. Two models representing opposite ends of the RLHF spectrum are tested: GPT-4o (closed-source, heavy alignment) and Llama-4-Scout (open-weight, lighter alignment), across three conditions (baseline, open-minded prompt, no initial vote), with N = 3 replications per cell (18 runs total). Three findings emerge. (i) Seventeen of eighteen runs end in a hung jury (a state where the jury fails to reach a unanimous verdict); the film's central event, gradual minority-to-majority persuasion, almost never occurs, indicating that anchoring is the dominant failure mode of current LLMs in this setting. (ii) The two models exhibit sharply different internal dynamics: GPT-4o produces a mean of 1.0 vote changes per run across all conditions, while Llama-4-Scout ranges from 2.0 (baseline) to 6.0 (open-minded prompt), and is the only model to reach a NOTGUILTY verdict (1 of 3 runs in the no-initial-vote condition). The same ``open-minded'' instruction is internalized by Llama and ignored by GPT-4o. (iii) This asymmetry suggests that the intensity of RLHF alignment training, not model capability, is the primary determinant of deliberative flexibility in multi-agent settings. Flexibility, not capability, tracks human deliberation. The work is framed as an exploratory study and discusses implications for jury-of-LLMs evaluation and multi-agent debate.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 9 of its 15 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

9 of 15 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May
  2. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  5. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  6. 2024Cognitive Bias in Decision-Making with LLMscited by 106
  7. 2023Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5cited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  2. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  3. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
  4. 2024Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debate
  5. 2026Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
  6. 2024Systematic Biases in LLM Simulations of Debates
  7. 2026From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
  8. 2024Persona Inconstancy in Multi-Agent LLM Collaboration: Conformity, Confabulation, and Impersonation