mlsanityData to 7 Oct 2026

Scaling Laws for Imitation Learning in Single-Agent Games

arXiv 2307.09423 ↗First posted 18 Jul 2023cs.LG cs.AI stat.MLcited by 11

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Imitation Learning (IL) is one of the most widely used methods in machine learning. Yet, many works find it is often unable to fully recover the underlying expert behavior, even in constrained environments like single-agent games. However, none of these works deeply investigate the role of scaling up the model and data size. Inspired by recent work in Natural Language Processing (NLP) where "scaling up" has resulted in increasingly more capable LLMs, we investigate whether carefully scaling up model and data size can bring similar improvements in the imitation learning setting for single-agent games. We first demonstrate our findings on a variety of Atari games, and thereafter focus on the extremely challenging game of NetHack. In all games, we find that IL loss and mean return scale smoothly with the compute budget (FLOPs) and are strongly correlated, resulting in power laws for training compute-optimal IL agents. Finally, we forecast and train several NetHack agents with IL and find they outperform prior state-of-the-art by 1.5x in all settings. Our work both demonstrates the scaling behavior of imitation learning in a variety of single-agent games, as well as the viability of scaling up current approaches for increasingly capable agents in NetHack, a game that remains elusively hard for current AI systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 40 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

30 of 40 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2023Scaling Data-Constrained Language Modelscited by 391
  10. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  12. 2019Causal Confusion in Imitation Learningcited by 286
18 more
  1. 2021Scaling Laws for Transfercited by 265
  2. 2020First return, then explorecited by 256
  3. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  4. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  5. 2020The NetHack Learning Environmentcited by 196
  6. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  7. 2022Broken Neural Scaling Lawscited by 102
  8. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  9. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  10. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  11. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  12. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  13. 2023Scaling laws for single-agent reinforcement learningcited by 39
  14. 2021Scaling Laws for Acoustic Modelscited by 20
  15. 2022Insights From the NeurIPS 2021 NetHack Challengecited by 20
  16. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  17. 2023NetHack is Hard to Hackcited by 11
  18. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 3.

11 papers.

  1. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  2. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  3. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  4. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  5. 2025Pixels to Play: A Foundation Model for 3D Gameplaycited by 3
  6. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  7. 2025Complexity Scaling Laws for Neural Models using Combinatorial Optimizationcited by 1
  8. 2025Learning to play: A Multimodal Agent for 3D Game-Playcited by 1
  9. 2024AI-Guided Molecular Simulations in VR: Exploring Strategies for Imitation Learning in Hyperdimensional Molecular Systemscited by 0
  10. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  11. 2026Up and Down the Abstraction Ladder: Code-Based Skills for Language Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Model
  2. 2026Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playing
  3. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation
  4. 2019Towards Interactive Training of Non-Player Characters in Video Games
  5. 2022Multi-Game Decision Transformers
  6. 2023Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games
  7. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
  8. 2022A Ranking Game for Imitation Learning