mlsanityData to 7 Oct 2026

AvalonBench: Evaluating LLMs Playing the Game of Avalon

arXiv 2310.05036 ↗First posted 8 Oct 2023cs.AI cs.CLcited by 45

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

In this paper, we explore the potential of Large Language Models (LLMs) Agents in playing the strategic social deduction game, Resistance Avalon. Players in Avalon are challenged not only to make informed decisions based on dynamically evolving game phases, but also to engage in discussions where they must deceive, deduce, and negotiate with other players. These characteristics make Avalon a compelling test-bed to study the decision-making and language-processing capabilities of LLM Agents. To facilitate research in this line, we introduce AvalonBench - a comprehensive game environment tailored for evaluating multi-agent LLM Agents. This benchmark incorporates: (1) a game environment for Avalon, (2) rule-based bots as baseline opponents, and (3) ReAct-style LLM agents with tailored prompts for each role. Notably, our evaluations based on AvalonBench highlight a clear capability gap. For instance, models like ChatGPT playing good-role got a win rate of 22.2% against rule-based bots playing evil, while good-role bot achieves 38.2% win rate in the same setting. We envision AvalonBench could be a good test-bed for developing more advanced LLMs (with self-playing) and agent frameworks that can effectively model the layered complexities of such game environments.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 24 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

16 of 24 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
4 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2023AVIS: Autonomous Visual Information Seeking with Large Language Model Agentcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 14; 2026: 18.

45 papers.

Sort:

By area:

  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  7. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  8. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 89
  9. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  10. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  11. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  12. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
33 more
  1. 2024PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Modelscited by 25
  2. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 25
  3. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  4. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 16
  5. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  7. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 12
  8. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  9. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  10. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9
  11. 2024Affordable Generative Agentscited by 8
  12. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  13. 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluationcited by 7
  14. 2025Time to Talk: LLM Agents for Asynchronous Group Communication in Mafia Gamescited by 5
  15. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMscited by 5
  16. 2025Zeus: Zero-shot LLM Instruction for Union Segmentation in Multimodal Medical Imagingcited by 4
  17. 2025MAPLE: Multi-Agent Adaptive Planning with Long-Term Memory for Table Reasoningcited by 4
  18. 2025The Stackelberg Speaker: Optimizing Persuasive Communication in Social Deduction Gamescited by 3
  19. 2026Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargainingcited by 3
  20. 2025InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Stylescited by 2
  21. 2026Beyond Preset Identities: How Agents Form Stances and Boundaries in Generative Societiescited by 2
  22. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  23. 2026AI for Games in the Foundation Model Eracited by 2
  24. 2025WGSR-Bench: Wargame-based Game-theoretic Strategic Reasoning Benchmark for Large Language Modelscited by 1
  25. 2025CogMem: A Cognitive Memory Architecture for Sustained Multi-Turn Reasoning in Large Language Modelscited by 1
  26. 2026Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournamentscited by 1
  27. 2026Foresight Optimization for Strategic Reasoning in Large Language Modelscited by 0
  28. 2026SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systemscited by 0
  29. 2026RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Modelscited by 0
  30. 2026CaM-Wolf: Causal-Aware Multimodal Agents for Social Deduction Gamescited by 0
  31. 2026Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanicscited by 0
  32. 2026MARBO: Relational Belief Grounding for LLM Agents in Social Deduction Gamescited by 0
  33. 2026Shockingly Simple Self-retrospection Improves Agentic Models Without RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay
  2. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
  3. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  4. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
  5. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration
  6. 2025Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity?
  7. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
  8. 2026SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models