# Scaling Laws for Imitation Learning in Single-Agent Games

arXiv: https://arxiv.org/abs/2307.09423 · HTML: https://mlsanity.com/paper/2307.09423/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Fighting Copycat Agents in Behavioral Cloning from Observation Histories](https://mlsanity.com/paper/2010.14876/) (2020)
- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [Atari-5: Distilling the Arcade Learning Environment down to Five Games](https://mlsanity.com/paper/2210.02019/) (2022)
- [Scaling laws for single-agent reinforcement learning](https://mlsanity.com/paper/2301.13442/) (2023)
- [Scaling Laws for Acoustic Models](https://mlsanity.com/paper/2106.09488/) (2021)
- [Insights From the NeurIPS 2021 NetHack Challenge](https://mlsanity.com/paper/2203.11889/) (2022)
- [Dungeons and Data: A Large-Scale NetHack Dataset](https://mlsanity.com/paper/2211.00539/) (2022)
- [NetHack is Hard to Hack](https://mlsanity.com/paper/2305.19240/) (2023)
- [Accelerating exploration and representation learning with offline pre-training](https://mlsanity.com/paper/2304.00046/) (2023)

## Built on by (most cited first, 11 of 11)

- [Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective](https://mlsanity.com/paper/2412.14135/) (2024)
- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [Scaling Laws for Pre-training Agents and World Models](https://mlsanity.com/paper/2411.04434/) (2024)
- [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025)
- [Pixels to Play: A Foundation Model for 3D Gameplay](https://mlsanity.com/paper/2508.14295/) (2025)
- [Survival Reinforcement Learning: Toward Scalable Self-Supervised RL](https://mlsanity.com/paper/2605.31273/) (2026)
- [Complexity Scaling Laws for Neural Models using Combinatorial Optimization](https://mlsanity.com/paper/2506.12932/) (2025)
- [Learning to play: A Multimodal Agent for 3D Game-Play](https://mlsanity.com/paper/2510.16774/) (2025)
- [AI-Guided Molecular Simulations in VR: Exploring Strategies for Imitation Learning in Hyperdimensional Molecular Systems](https://mlsanity.com/paper/2409.07189/) (2024)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents](https://mlsanity.com/paper/2609.31076/) (2026)

## Abstract (arXiv)

Imitation Learning (IL) is one of the most widely used methods in machine learning. Yet, many works find it is often unable to fully recover the underlying expert behavior, even in constrained environments like single-agent games. However, none of these works deeply investigate the role of scaling up the model and data size. Inspired by recent work in Natural Language Processing (NLP) where "scaling up" has resulted in increasingly more capable LLMs, we investigate whether carefully scaling up model and data size can bring similar improvements in the imitation learning setting for single-agent games. We first demonstrate our findings on a variety of Atari games, and thereafter focus on the extremely challenging game of NetHack. In all games, we find that IL loss and mean return scale smoothly with the compute budget (FLOPs) and are strongly correlated, resulting in power laws for training compute-optimal IL agents. Finally, we forecast and train several NetHack agents with IL and find they outperform prior state-of-the-art by 1.5x in all settings. Our work both demonstrates the scaling behavior of imitation learning in a variety of single-agent games, as well as the viability of scaling up current approaches for increasingly capable agents in NetHack, a game that remains elusively hard for current AI systems.