mlsanityData to 14 Oct 2026

Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents

arXiv 2305.13455 ↗First posted 22 May 2023cs.CLcited by 54

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent work has proposed a methodology for the systematic evaluation of "Situated Language Understanding Agents"-agents that operate in rich linguistic and non-linguistic contexts-through testing them in carefully constructed interactive settings. Other recent work has argued that Large Language Models (LLMs), if suitably set up, can be understood as (simulators of) such agents. A connection suggests itself, which this paper explores: Can LLMs be evaluated meaningfully by exposing them to constrained game-like settings that are built to challenge specific capabilities? As a proof of concept, this paper investigates five interaction settings, showing that current chat-optimised LLMs are, to an extent, capable to follow game-play instructions. Both this capability and the quality of the game play, measured by how well the objectives of the different games are met, follows the development cycle, with newer models performing better. The metrics even for the comparatively simple example games are far from being saturated, suggesting that the proposed instrument will remain to have diagnostic value. Our general framework for implementing and evaluating games with LLMs is available at https://github.com/clembench .

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 44 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

20 of 44 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
8 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  2. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  3. 2022Language Models as Agent Modelscited by 132
  4. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  5. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12
  6. 2023What A Situated Language-Using Agent Must be Able to Do: A Top-Down Analysiscited by 8
  7. 2022The slurk Interaction Server Framework: Better Data for Better Dialog Modelscited by 7
  8. 2022Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledgecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 16; 2025: 15; 2026: 17.

54 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  3. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  4. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  5. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  6. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 95
  7. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 94
  8. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  9. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  10. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  11. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agentscited by 28
  12. 2025TextArenacited by 27
38 more
  1. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  2. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  3. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 15
  4. 2024Evaluating Dialect Robustness of Language Models via Conversation Understandingcited by 14
  5. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 12
  6. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agentscited by 8
  7. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  8. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  9. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  10. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  11. 2024Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Modelscited by 6
  12. 2025clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisationscited by 4
  13. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 3
  14. 2025A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembenchcited by 3
  15. 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
  16. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  17. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  18. 2024Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programmingcited by 2
  19. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  20. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  21. 2025Do You Get the Hint? Benchmarking LLMs on the Board Game Conceptcited by 2
  22. 2025Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Modelscited by 1
  23. 2025Plant in Cupboard, Orange on Rably, Inat Aphone. Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environmentcited by 1
  24. 2025Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialoguescited by 1
  25. 2025Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilitiescited by 1
  26. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
  27. 2023Leveraging Large Language Model for Automatic Evolving of Industrial Data-Centric R&D Cyclecited by 0
  28. 2023On General Language Understandingcited by 0
  29. 2024How Many Parameters Does it Take to Change a Light Bulb? Evaluating Performance in Self-Play of Conversational Games as a Function of Model Characteristicscited by 0
  30. 2024Strategic Insights in Human and Large Language Model Tactics at Word Guessing Gamescited by 0
  31. 2025PARL: Prompt-based Agents for Reinforcement Learningcited by 0
  32. 2025MARIA: A Framework for Marginal Risk Assessment without Ground Truth in AI Systemscited by 0
  33. 2026The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialoguecited by 0
  34. 2026Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Gamescited by 0
  35. 2026Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+cited by 0
  36. 2026Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inferencecited by 0
  37. 2026Investigating Knowledge Transfer Across Interactive Dialogue Gamescited by 0
  38. 2026First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agentscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions
  2. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  3. 2023AgentBench: Evaluating LLMs as Agents
  4. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios
  5. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  6. 2026Playing With AI: How Do State-Of-The-Art Large Language Models Perform in the 1977 Text-Based Adventure Game Zork?
  7. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  8. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey