mlsanityData to 14 Oct 2026

How Many Parameters Does it Take to Change a Light Bulb? Evaluating Performance in Self-Play of Conversational Games as a Function of Model Characteristics

arXiv 2406.14051 ↗First posted 20 Jun 2024cs.CL cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning

Abstract (arXiv)

What makes a good Large Language Model (LLM)? That it performs well on the relevant benchmarks -- which hopefully measure, with some validity, the presence of capabilities that are also challenged in real application. But what makes the model perform well? What gives a model its abilities? We take a recently introduced type of benchmark that is meant to challenge capabilities in a goal-directed, agentive context through self-play of conversational games, and analyse how performance develops as a function of model characteristics like number of parameters, or type of training. We find that while there is a clear relationship between number of parameters and performance, there is still a wide spread of performance points within a given size bracket, which is to be accounted for by training parameters such as fine-tuning data quality and method. From a more practical angle, we also find a certain degree of unpredictability about performance across access methods, possible due to unexposed sampling parameters, and a, very welcome, performance stability against at least moderate weight quantisation during inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 20 of its 27 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

20 of 27 references.

References · 20

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  8. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  9. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  11. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  12. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
8 more
  1. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  2. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  3. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  4. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  5. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 54
  6. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 37
  7. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  8. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agentscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures
  2. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  3. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  4. 2025The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
  5. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  6. 2023Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games
  7. 2026S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
  8. 2026Choosing LLMs for Strategic Agent Interactions