mlsanityData to 14 Oct 2026

Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction

arXiv 2407.13943 ↗First posted 18 Jul 2024cs.CL cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

This paper introduces Werewolf Arena, a novel framework for evaluating large language models (LLMs) through the lens of the classic social deduction game, Werewolf. In Werewolf Arena, LLMs compete against each other, navigating the game's complex dynamics of deception, deduction, and persuasion. The framework introduces a dynamic turn-taking system based on bidding, mirroring real-world discussions where individuals strategically choose when to speak. We demonstrate the framework's utility through an arena-style tournament featuring Gemini and GPT models. Our results reveal distinct strengths and weaknesses in the models' strategic reasoning and communication. These findings highlight Werewolf Arena's potential as a challenging and scalable LLM benchmark.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

21 of 34 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  6. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  7. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  8. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  9. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  10. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 209
  11. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  12. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 126
9 more
  1. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  2. 2018Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learningcited by 53
  3. 2021MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understandingcited by 36
  4. 2024Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolfcited by 24
  5. 2024Simulating Human Strategic Behavior: Comparing Single and Multi-agent LLMscited by 21
  6. 2023Playing the Werewolf game with artificial intelligence for language understandingcited by 16
  7. 2022Putting the Con in Context: Identifying Deceptive Actors in the Game of Mafiacited by 13
  8. 2023Is ChatGPT a Good Multi-Party Conversation Solver?cited by 13
  9. 2022Werewolf Among Us: A Multimodal Dataset for Modeling Persuasion Behaviors in Social Deduction Gamescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 24.

45 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 31
  3. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  4. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 23
  5. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 17
  6. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  7. 2025Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimizationcited by 15
  8. 2025WOLF: Werewolf-based Observations for LLM Deception and Falsehoodscited by 13
  9. 2025SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environmentscited by 11
  10. 2025Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Gamecited by 10
  11. 2025Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategiescited by 10
  12. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
33 more
  1. 2025Deceive, Detect, and Disclose: Large Language Models Play Mini-Mafiacited by 9
  2. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  3. 2024Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Gamescited by 7
  4. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  5. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  6. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMscited by 5
  7. 2026PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solverscited by 5
  8. 2025PuzzleJAX: A Benchmark for Reasoning and Learningcited by 4
  9. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  10. 2025TongSIM: A General Platform for Simulating Intelligent Machinescited by 4
  11. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  12. 2026QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agentscited by 3
  13. 2026AI for Games in the Foundation Model Eracited by 3
  14. 2025Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Marketscited by 2
  15. 2025Strategy Adaptation in Large Language Model Werewolf Agentscited by 2
  16. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  17. 2025Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Modelscited by 2
  18. 2026Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Gamescited by 2
  19. 2026MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Gamescited by 2
  20. 2026Evaluating Large Language Models in a Complex Hidden Role Gamecited by 1
  21. 2026RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialoguecited by 1
  22. 2026Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Gamecited by 0
  23. 2026AIT Academy: Cultivating the Complete Agent with a Confucian Three-Domain Curriculumcited by 0
  24. 2026Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Playcited by 0
  25. 2026Triadic Werewolf: A Jester Role for Multi-Hop Theory of Mind in LLMscited by 0
  26. 2026Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Gamescited by 0
  27. 2026Even More Deception: Objective Misalignment in Mixed-Motive LLM Multi-Agent Systemscited by 0
  28. 2026Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Gamecited by 0
  29. 2026Persuasive and Compliant Tendencies Predict Group Decision-Making in Humans and Language Modelscited by 0
  30. 2026Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Modelscited by 0
  31. 2026MARBO: Relational Belief Grounding for LLM Agents in Social Deduction Gamescited by 0
  32. 2026Playing social deduction games with reinforcement fine-tuned large language modelscited by 0
  33. 2026Communication Shapes Collective Inference in Self-Adapting LLM Societies: Evidence from Mafiacited by 0