mlsanityData to 14 Oct 2026

Playpen: An Environment for Exploring Learning Through Conversational Interaction

arXiv 2504.08590 ↗First posted 11 Apr 2025cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appropriateness of a model's response. In this paper, we investigate whether Dialogue Games -- goal-directed and rule-governed activities driven predominantly by verbal actions -- can also serve as a source of feedback signals for learning. We introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning; direct alignment (DPO); and reinforcement learning with GRPO. We experiment with post-training a small LLM (Llama-3.1-8B-Instruct), evaluating performance on unseen instances of training games as well as unseen games, and on standard benchmarks. We find that imitation learning through SFT improves performance on unseen instances, but negatively impacts other skills, while interactive learning with GRPO shows balanced improvements without loss of skills. We release the framework and the baseline training setups to foster research in the promising new direction of learning in (synthetic) interaction.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 107 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

66 of 107 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF6 Oct
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2020Learning Rewards from Linguistic Feedback30 Sep

Other references · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  7. 2023LIMA: Less Is More for Alignmentcited by 1,100
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  9. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  10. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  11. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  12. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  13. 2024Iterative Reasoning Preference Optimizationcited by 246
  14. 2020Experience Grounds Languagecited by 242
  15. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  16. 2020The NetHack Learning Environmentcited by 198
  17. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  18. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  19. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  20. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  21. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
  22. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  23. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  24. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  25. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 107
  26. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  27. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  28. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  29. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  30. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 80
  31. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 70
  32. 2018Emergent Communication through Negotiationcited by 62
  33. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  34. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 57
  35. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 54
  36. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 37
  37. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  38. 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  2. 2025A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembenchcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agents
  2. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
  3. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  4. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
  5. 2025Language Self-Play For Data-Free Training
  6. 2025Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
  7. 2025PRINCIPLES: Synthetic Strategy Memory for Proactive Dialogue Agents
  8. 2020Human-centric Dialog Training via Offline Reinforcement Learning