# Learning Controllable Content Generators

arXiv: https://arxiv.org/abs/2105.02993 · HTML: https://mlsanity.com/paper/2105.02993/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [Learning to Navigate in Cities Without a Map](https://mlsanity.com/paper/1804.00168/) (2018)
- [PCGRL: Procedural Content Generation via Reinforcement Learning](https://mlsanity.com/paper/2001.09212/) (2020)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation](https://mlsanity.com/paper/1806.10729/) (2018)
- [Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Network](https://mlsanity.com/paper/1805.00728/) (2018)
- [Deep Learning for Procedural Content Generation](https://mlsanity.com/paper/2010.04548/) (2020)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithms](https://mlsanity.com/paper/1802.10363/) (2018)
- [Adversarial Reinforcement Learning for Procedural Content Generation](https://mlsanity.com/paper/2103.04847/) (2021)
- [Mixed-Initiative Level Design with RL Brush](https://mlsanity.com/paper/2008.02778/) (2020)
- [Using Fractal Neural Networks to Play SimCity 1 and Conway's Game of Life at Variable Scales](https://mlsanity.com/paper/2002.03896/) (2020)

## Built on by (most cited first, 30 of 40)

- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Level Generation Through Large Language Models](https://mlsanity.com/paper/2302.05817/) (2023)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [Deep Surrogate Assisted Generation of Environments](https://mlsanity.com/paper/2206.04199/) (2022)
- [Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting](https://mlsanity.com/paper/2411.09823/) (2024)
- [Illuminating Diverse Neural Cellular Automata for Level Generation](https://mlsanity.com/paper/2109.05489/) (2021)
- [Learning Controllable 3D Level Generators](https://mlsanity.com/paper/2206.13623/) (2022)
- [Path of Destruction: Learning an Iterative Level Generator Using a Small Dataset](https://mlsanity.com/paper/2202.10184/) (2022)
- [Games for Artificial Intelligence Research: A Review and Perspectives](https://mlsanity.com/paper/2304.13269/) (2023)
- [DreamGarden: A Designer Assistant for Growing Games from a Single Prompt](https://mlsanity.com/paper/2410.01791/) (2024)
- [Mutation Models: Learning to Generate Levels by Imitating Evolution](https://mlsanity.com/paper/2206.05497/) (2022)
- [RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss Raid Games](https://mlsanity.com/paper/2307.01676/) (2023)
- [PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators](https://mlsanity.com/paper/2408.12525/) (2024)
- [Latent Prompt Tuning for Text Summarization](https://mlsanity.com/paper/2211.01837/) (2022)
- [Arachnophobia Exposure Therapy using Experience-driven Procedural Content Generation via Reinforcement Learning (EDPCGRL)](https://mlsanity.com/paper/2110.04146/) (2021)
- [Measuring Diversity of Game Scenarios](https://mlsanity.com/paper/2404.15192/) (2024)
- [Hierarchically Composing Level Generators for the Creation of Complex Structures](https://mlsanity.com/paper/2302.01561/) (2023)
- [Balancing of competitive two-player Game Levels with Reinforcement Learning](https://mlsanity.com/paper/2306.04429/) (2023)
- [G-PCGRL: Procedural Graph Data Generation via Reinforcement Learning](https://mlsanity.com/paper/2407.10483/) (2024)
- [IPCGRL: Language-Instructed Reinforcement Learning for Procedural Level Generation](https://mlsanity.com/paper/2503.12358/) (2025)
- [PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning](https://mlsanity.com/paper/2502.10906/) (2025)
- [Multi-Objective Instruction-Aware Representation Learning in Procedural Content Generation RL](https://mlsanity.com/paper/2508.09193/) (2025)
- [Adapting Procedural Content Generation to Player Personas Through Evolution](https://mlsanity.com/paper/2112.04406/) (2021)
- [Online Game Level Generation from Music](https://mlsanity.com/paper/2207.05271/) (2022)
- [Aesthetic Bot: Interactively Evolving Game Maps on Twitter](https://mlsanity.com/paper/2208.05017/) (2022)
- [Start Small: Training Controllable Game Level Generators without Training Data by Learning at Multiple Sizes](https://mlsanity.com/paper/2209.15052/) (2022)
- [Human-Aligned Procedural Level Generation Reinforcement Learning via Text-Level-Sketch Shared Representation](https://mlsanity.com/paper/2508.09860/) (2025)
- [State Space Closure: Revisiting Endless Online Level Generation via Reinforcement Learning](https://mlsanity.com/paper/2212.02951/) (2022)
- [Controllable Path of Destruction](https://mlsanity.com/paper/2305.18553/) (2023)

## Abstract (arXiv)

It has recently been shown that reinforcement learning can be used to train generators capable of producing high-quality game levels, with quality defined in terms of some user-specified heuristic. To ensure that these generators' output is sufficiently diverse (that is, not amounting to the reproduction of a single optimal level configuration), the generation process is constrained such that the initial seed results in some variance in the generator's output. However, this results in a loss of control over the generated content for the human user. We propose to train generators capable of producing controllably diverse output, by making them "goal-aware." To this end, we add conditional inputs representing how close a generator is to some heuristic, and also modify the reward mechanism to incorporate that value. Testing on multiple domains, we show that the resulting level generators are capable of exploring the space of possible levels in a targeted, controllable manner, producing levels of comparable quality as their goal-unaware counterparts, that are diverse along designer-specified dimensions.