mlsanityData to 7 Oct 2026

clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents

arXiv 2405.20859 ↗First posted 31 May 2024cs.CL cs.AIcited by 8

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

It has been established in recent work that Large Language Models (LLMs) can be prompted to "self-play" conversational games that probe certain capabilities (general instruction following, strategic goal orientation, language understanding abilities), where the resulting interactive game play can be automatically scored. In this paper, we take one of the proposed frameworks for setting up such game-play environments, and further test its usefulness as an evaluation instrument, along a number of dimensions: We show that it can easily keep up with new developments while avoiding data contamination, we show that the tests implemented within it are not yet saturated (human performance is substantially higher than that of even the best models), and we show that it lends itself to investigating additional questions, such as the impact of the prompting language on performance. We believe that the approach forms a good basis for making decisions on model choice for building applied interactive systems, and perhaps ultimately setting up a closed-loop development environment of system and simulated evaluator.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 16 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

10 of 16 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agents2 Oct
  2. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents22 May

Other references · 8

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  3. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  4. 2022Data Contamination: From Memorization to Exploitationcited by 234
  5. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  6. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  7. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  8. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  2. 2025A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembenchcited by 3
  3. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 2
  4. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  5. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  6. 2025Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Modelscited by 1
  7. 2024How Many Parameters Does it Take to Change a Light Bulb? Evaluating Performance in Self-Play of Conversational Games as a Function of Model Characteristicscited by 0
  8. 2026First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  3. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  4. 2024A Survey on Large Language Model-Based Game Agents
  5. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
  6. 2026GameTalk: Training LLMs for Strategic Conversation
  7. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  8. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments