WiS Platform: Enhancing Evaluation of LLM-Based Multi-Agent Systems Through Game-Based Analysis
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Recent advancements in autonomous multi-agent systems (MAS) based on large language models (LLMs) have enhanced the application scenarios and improved the capability of LLMs to handle complex tasks. Despite demonstrating effectiveness, existing studies still evidently struggle to evaluate, analysis, and reproducibility of LLM-based MAS. In this paper, to facilitate the research on LLM-based MAS, we introduce an open, scalable, and real-time updated platform for accessing and analyzing the LLM-based MAS based on the games Who is Spy?" (WiS). Our platform is featured with three main worths: (1) a unified model evaluate interface that supports models available on Hugging Face; (2) real-time updated leaderboard for model evaluation; (3) a comprehensive evaluation covering game-winning rates, attacking, defense strategies, and reasoning of LLMs. To rigorously test WiS, we conduct extensive experiments coverage of various open- and closed-source LLMs, we find that different agents exhibit distinct and intriguing behaviors in the game. The experimental results demonstrate the effectiveness and efficiency of our platform in evaluating LLM-based MAS. Our platform and its documentation are publicly available at https://whoisspy.ai/.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 29 references that are in the map. First posted on arXiv 4 Dec 2024.
Builds on
17 of 29 references.
Closest ancestors · 3
Same direction, newest first
- 2023Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game29 Dec
- 2023Lyfe Agents: Generative agents for low-cost real-time social interactions3 Oct
- 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug
Other references · 14
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023ChatDev: Communicative Agents for Software Developmentcited by 893
- 2023Self-collaboration Code Generation via ChatGPTcited by 255
- 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
- 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
- 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
- 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
- 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
- 2021Chess as a Testbed for Language Model State Trackingcited by 69
- 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
- 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Makingcited by 22
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Game-Theoretic Lens on LLM-based Multi-Agent Systems
- 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application
- 2025Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs
- 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
- 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
- 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems