# PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators

arXiv: https://arxiv.org/abs/2408.12525 · HTML: https://mlsanity.com/paper/2408.12525/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [PCGRL: Procedural Content Generation via Reinforcement Learning](https://mlsanity.com/paper/2001.09212/) (2020)
- [Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2402.16801/) (2024)
- [Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Network](https://mlsanity.com/paper/1805.00728/) (2018)
- [Deep Learning for Procedural Content Generation](https://mlsanity.com/paper/2010.04548/) (2020)
- [Super Mario as a String: Platformer Level Generation Via LSTMs](https://mlsanity.com/paper/1603.00930/) (2016)
- [Learning Controllable Content Generators](https://mlsanity.com/paper/2105.02993/) (2021)
- [Rotation, Translation, and Cropping for Zero-Shot Generalization](https://mlsanity.com/paper/2001.09908/) (2020)
- [Learning Controllable 3D Level Generators](https://mlsanity.com/paper/2206.13623/) (2022)
- [Co-Creative Level Design via Machine Learning](https://mlsanity.com/paper/1809.09420/) (2018)
- [Mixed-Initiative Level Design with RL Brush](https://mlsanity.com/paper/2008.02778/) (2020)
- [Addressing the Fundamental Tension of PCGML with Discriminative Learning](https://mlsanity.com/paper/1809.04432/) (2018)

## Built on by (most cited first, 7 of 7)

- [IPCGRL: Language-Instructed Reinforcement Learning for Procedural Level Generation](https://mlsanity.com/paper/2503.12358/) (2025)
- [PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning](https://mlsanity.com/paper/2502.10906/) (2025)
- [Multi-Objective Instruction-Aware Representation Learning in Procedural Content Generation RL](https://mlsanity.com/paper/2508.09193/) (2025)
- [Human-Aligned Procedural Level Generation Reinforcement Learning via Text-Level-Sketch Shared Representation](https://mlsanity.com/paper/2508.09860/) (2025)
- [Video Game Level Design as a Multi-Agent Reinforcement Learning Problem](https://mlsanity.com/paper/2510.04862/) (2025)
- [Level the Level: Balancing Game Levels for Asymmetric Player Archetypes With Reinforcement Learning](https://mlsanity.com/paper/2503.24099/) (2025)
- [An Exploration of Collision-based Enemy Morphology Generation](https://mlsanity.com/paper/2606.02832/) (2026)

## Abstract (arXiv)

Procedural Content Generation via Reinforcement Learning (PCGRL) has been introduced as a means by which controllable designer agents can be trained based only on a set of computable metrics acting as a proxy for the level's quality and key characteristics. While PCGRL offers a unique set of affordances for game designers, it is constrained by the compute-intensive process of training RL agents, and has so far been limited to generating relatively small levels. To address this issue of scale, we implement several PCGRL environments in Jax so that all aspects of learning and simulation happen in parallel on the GPU, resulting in faster environment simulation; removing the CPU-GPU transfer of information bottleneck during RL training; and ultimately resulting in significantly improved training speed. We replicate several key results from prior works in this new framework, letting models train for much longer than previously studied, and evaluating their behavior after 1 billion timesteps. Aiming for greater control for human designers, we introduce randomized level sizes and frozen "pinpoints" of pivotal game tiles as further ways of countering overfitting. To test the generalization ability of learned generators, we evaluate models on large, out-of-distribution map sizes, and find that partial observation sizes learn more robust design strategies.