mlsanityData to 7 Oct 2026

GameEval: Evaluating LLMs on Conversational Games

arXiv 2308.10032 ↗First posted 19 Aug 2023cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

The rapid advancements in large language models (LLMs) have presented challenges in evaluating those models. Existing evaluation methods are either reference-based or preference based, which inevitably need human intervention or introduce test bias caused by evaluator models. In this paper, we propose GameEval, a novel approach to evaluating LLMs through goal-driven conversational games, overcoming the limitations of previous methods. GameEval treats LLMs as game players and assigns them distinct roles with specific goals achieved by launching conversations of various forms, including discussion, question answering, and voting. We design three unique games with cooperative or adversarial objectives, accompanied by corresponding evaluation metrics, to show how this new paradigm comprehensively evaluates model performance.Through extensive experiments, we show that GameEval can effectively differentiate the capabilities of various LLMs, providing a comprehensive assessment of their integrated abilities to solve complex problems. Our public anonymous code is available at https://github.com/GameEval/GameEval.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 19 Aug 2023.

Builds on

20 of 28 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
8 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  3. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  4. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  5. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  6. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  7. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  8. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 16; 2025: 11; 2026: 5.

35 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  2. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  3. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  4. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  5. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  6. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environmentscited by 35
  7. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 25
  8. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  9. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  10. 2024Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolfcited by 22
  11. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  12. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
23 more
  1. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14
  2. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  3. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  4. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  5. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agentscited by 8
  6. 2023Leveraging Word Guessing Games to Assess the Intelligence of Large Language Modelscited by 7
  7. 2024EscapeBench: Towards Advancing Creative Intelligence of Language Model Agentscited by 5
  8. 2025clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisationscited by 4
  9. 202590% Faster, 100% Code-Free: MLLM-Driven Zero-Code 3D Game Developmentcited by 4
  10. 2024Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Gamecited by 3
  11. 2024Fine-Grained and Thematic Evaluation of LLMs in Social Deduction Gamecited by 3
  12. 2025A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembenchcited by 3
  13. 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
  14. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 2
  15. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  16. 2025Is Your LLM Really Mastering the Concept? A Multi-Agent Benchmarkcited by 1
  17. 2025FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCscited by 1
  18. 2026Skill Issue: Are Skills Language-Invariant in LLMs?cited by 1
  19. 2024How Many Parameters Does it Take to Change a Light Bulb? Evaluating Performance in Self-Play of Conversational Games as a Function of Model Characteristicscited by 0
  20. 2024WiS Platform: Enhancing Evaluation of LLM-Based Multi-Agent Systems Through Game-Based Analysiscited by 0
  21. 2026Inferring Latent Intentions: Attributional Natural Language Inference in LLM Agentscited by 0
  22. 2026Multi-Turn Multi-Agent Dialogue for Collaborative Reconstruction Improves VLM Performance on Spatial Reasoning, But Only Barelycited by 0
  23. 2026Investigating Knowledge Transfer Across Interactive Dialogue Gamescited by 0