# Adversarial Reinforcement Learning for Procedural Content Generation

arXiv: https://arxiv.org/abs/2103.04847 · HTML: https://mlsanity.com/paper/2103.04847/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Assessing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/1810.12282/) (2018)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [PCGRL: Procedural Content Generation via Reinforcement Learning](https://mlsanity.com/paper/2001.09212/) (2020)
- [Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation](https://mlsanity.com/paper/1806.10729/) (2018)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [Imitation Learning with Concurrent Actions in 3D Games](https://mlsanity.com/paper/1803.05402/) (2018)

## Built on by (most cited first, 30 of 36)

- [Learning Controllable Content Generators](https://mlsanity.com/paper/2105.02993/) (2021)
- [Direct Behavior Specification via Constrained Reinforcement Learning](https://mlsanity.com/paper/2112.12228/) (2021)
- [Experience-Driven PCG via Reinforcement Learning: A Super Mario Bros Study](https://mlsanity.com/paper/2106.15877/) (2021)
- [Reinforcement Learning for Education: Opportunities and Challenges](https://mlsanity.com/paper/2107.08828/) (2021)
- [DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraft](https://mlsanity.com/paper/2404.15538/) (2024)
- [Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning](https://mlsanity.com/paper/2208.07811/) (2022)
- [Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning](https://mlsanity.com/paper/2105.14780/) (2021)
- [Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation](https://mlsanity.com/paper/2207.08379/) (2022)
- [RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss Raid Games](https://mlsanity.com/paper/2307.01676/) (2023)
- [CCPT: Automatic Gameplay Testing and Validation with Curiosity-Conditioned Proximal Trajectories](https://mlsanity.com/paper/2202.10057/) (2022)
- [From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving](https://mlsanity.com/paper/2305.13960/) (2023)
- [Arachnophobia Exposure Therapy using Experience-driven Procedural Content Generation via Reinforcement Learning (EDPCGRL)](https://mlsanity.com/paper/2110.04146/) (2021)
- [Graph augmented Deep Reinforcement Learning in the GameRLand3D environment](https://mlsanity.com/paper/2112.11731/) (2021)
- [Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policy](https://mlsanity.com/paper/2511.09737/) (2025)
- [Godot Reinforcement Learning Agents](https://mlsanity.com/paper/2112.03636/) (2021)
- [Hierarchically Composing Level Generators for the Creation of Complex Structures](https://mlsanity.com/paper/2302.01561/) (2023)
- [Balancing of competitive two-player Game Levels with Reinforcement Learning](https://mlsanity.com/paper/2306.04429/) (2023)
- [G-PCGRL: Procedural Graph Data Generation via Reinforcement Learning](https://mlsanity.com/paper/2407.10483/) (2024)
- [Augmentative Topology Agents For Open-Ended Learning](https://mlsanity.com/paper/2210.11442/) (2022)
- [CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing](https://mlsanity.com/paper/2304.13616/) (2023)
- [Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning](https://mlsanity.com/paper/2505.08264/) (2025)
- [Adapting Procedural Content Generation to Player Personas Through Evolution](https://mlsanity.com/paper/2112.04406/) (2021)
- [Start Small: Training Controllable Game Level Generators without Training Data by Learning at Multiple Sizes](https://mlsanity.com/paper/2209.15052/) (2022)
- [PCGPT: Procedural Content Generation via Transformers](https://mlsanity.com/paper/2310.02405/) (2023)
- [Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art](https://mlsanity.com/paper/2311.10129/) (2023)
- [Fly, Fail, Fix: Iterative Game Repair with Reinforcement Learning and Large Multimodal Models](https://mlsanity.com/paper/2507.12666/) (2025)
- [AbideGym: Turning Static RL Worlds into Adaptive Challenges](https://mlsanity.com/paper/2509.21234/) (2025)
- [From World-Gen to Quest-Line: A Dependency-Driven Prompt Pipeline for Coherent RPG Generation](https://mlsanity.com/paper/2604.25482/) (2026)
- [The State-Action-Reward-State-Action Algorithm in Spatial Prisoner's Dilemma Game](https://mlsanity.com/paper/2406.17326/) (2024)
- [Improving Conditional Level Generation using Automated Validation in Match-3 Games](https://mlsanity.com/paper/2409.06349/) (2024)

## Abstract (arXiv)

We present a new approach ARLPCG: Adversarial Reinforcement Learning for Procedural Content Generation, which procedurally generates and tests previously unseen environments with an auxiliary input as a control variable. Training RL agents over novel environments is a notoriously difficult task. One popular approach is to procedurally generate different environments to increase the generalizability of the trained agents. ARLPCG instead deploys an adversarial model with one PCG RL agent (called Generator) and one solving RL agent (called Solver). The Generator receives a reward signal based on the Solver's performance, which encourages the environment design to be challenging but not impossible. To further drive diversity and control of the environment generation, we propose using auxiliary inputs for the Generator. The benefit is two-fold: Firstly, the Solver achieves better generalization through the Generator's generated challenges. Secondly, the trained Generator can be used as a creator of novel environments that, together with the Solver, can be shown to be solvable. We create two types of 3D environments to validate our model, representing two popular game genres: a third-person platformer and a racing game. In these cases, we shows that ARLPCG has a significantly better solve ratio, and that the auxiliary inputs renders the levels creation controllable to a certain degree. For a video compilation of the results please visit https://youtu.be/z7q2PtVsT0I.