# Experience-Driven PCG via Reinforcement Learning: A Super Mario Bros Study

arXiv: https://arxiv.org/abs/2106.15877 · HTML: https://mlsanity.com/paper/2106.15877/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [PCGRL: Procedural Content Generation via Reinforcement Learning](https://mlsanity.com/paper/2001.09212/) (2020)
- [Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Network](https://mlsanity.com/paper/1805.00728/) (2018)
- [Deep Learning for Procedural Content Generation](https://mlsanity.com/paper/2010.04548/) (2020)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithms](https://mlsanity.com/paper/1802.10363/) (2018)
- [Procedural Content Generation through Quality Diversity](https://mlsanity.com/paper/1907.04053/) (2019)
- [The VGLC: The Video Game Level Corpus](https://mlsanity.com/paper/1606.07487/) (2016)
- [Adversarial Reinforcement Learning for Procedural Content Generation](https://mlsanity.com/paper/2103.04847/) (2021)
- [Tile Pattern KL-Divergence for Analysing and Evolving Game Levels](https://mlsanity.com/paper/1905.05077/) (2019)
- [Quality Diversity Through Surprise](https://mlsanity.com/paper/1807.02397/) (2018)
- [A Novel CNet-assisted Evolutionary Level Repairer and Its Applications to Super Mario Bros](https://mlsanity.com/paper/2005.06148/) (2020)

## Built on by (most cited first, 25 of 25)

- [Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2604.20987/) (2026)
- [Games for Artificial Intelligence Research: A Review and Perspectives](https://mlsanity.com/paper/2304.13269/) (2023)
- [Affective Game Computing: A Survey](https://mlsanity.com/paper/2309.14104/) (2023)
- [Mutation Models: Learning to Generate Levels by Imitating Evolution](https://mlsanity.com/paper/2206.05497/) (2022)
- [Open-Ended Evolution for Minecraft Building Generation](https://mlsanity.com/paper/2209.03108/) (2022)
- [Generative Personas That Behave and Experience Like Humans](https://mlsanity.com/paper/2209.00459/) (2022)
- [RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss Raid Games](https://mlsanity.com/paper/2307.01676/) (2023)
- [Go-Blend behavior and affect](https://mlsanity.com/paper/2109.13388/) (2021)
- [Measuring Diversity of Game Scenarios](https://mlsanity.com/paper/2404.15192/) (2024)
- [Hierarchically Composing Level Generators for the Creation of Complex Structures](https://mlsanity.com/paper/2302.01561/) (2023)
- [Keiki: Towards Realistic Danmaku Generation via Sequential GANs](https://mlsanity.com/paper/2107.02991/) (2021)
- [Adapting Procedural Content Generation to Player Personas Through Evolution](https://mlsanity.com/paper/2112.04406/) (2021)
- [Generating Game Levels of Diverse Behaviour Engagement](https://mlsanity.com/paper/2207.02100/) (2022)
- [Online Game Level Generation from Music](https://mlsanity.com/paper/2207.05271/) (2022)
- [Towards Objective Metrics for Procedurally Generated Video Game Levels](https://mlsanity.com/paper/2201.10334/) (2022)
- [Persona-driven Dominant/Submissive Map (PDSM) Generation for Tutorials](https://mlsanity.com/paper/2204.05217/) (2022)
- [State Space Closure: Revisiting Endless Online Level Generation via Reinforcement Learning](https://mlsanity.com/paper/2212.02951/) (2022)
- [Reconstructing Existing Levels through Level Inpainting](https://mlsanity.com/paper/2309.09472/) (2023)
- [Closing the Affective Loop via Experience-Driven Reinforcement Learning Designers](https://mlsanity.com/paper/2408.06346/) (2024)
- [Improving Deep Localized Level Analysis: How Game Logs Can Help](https://mlsanity.com/paper/2212.03376/) (2022)
- [Level Assembly as a Markov Decision Process](https://mlsanity.com/paper/2304.13922/) (2023)
- [RAMario: Experimental Approach to Reptile Algorithm -- Reinforcement Learning for Mario](https://mlsanity.com/paper/2305.09655/) (2023)
- [A Markovian Framing of WaveFunctionCollapse for Procedurally Generating Aesthetically Complex Environments](https://mlsanity.com/paper/2509.09919/) (2025)
- [Expanding Horizons of Level Diversity via Multi-objective Evolutionary Learning](https://mlsanity.com/paper/2509.24341/) (2025)
- [Learning Local Constraints for Reinforcement-Learned Content Generators](https://mlsanity.com/paper/2605.13570/) (2026)

## Abstract (arXiv)

We introduce a procedural content generation (PCG) framework at the intersections of experience-driven PCG and PCG via reinforcement learning, named ED(PCG)RL, EDRL in short. EDRL is able to teach RL designers to generate endless playable levels in an online manner while respecting particular experiences for the player as designed in the form of reward functions. The framework is tested initially in the Super Mario Bros game. In particular, the RL designers of Super Mario Bros generate and concatenate level segments while considering the diversity among the segments. The correctness of the generation is ensured by a neural net-assisted evolutionary level repairer and the playability of the whole level is determined through AI-based testing. Our agents in this EDRL implementation learn to maximise a quantification of Koster's principle of fun by moderating the degree of diversity across level segments. Moreover, we test their ability to design fun levels that are diverse over time and playable. Our proposed framework is capable of generating endless, playable Super Mario Bros levels with varying degrees of fun, deviation from earlier segments, and playability. EDRL can be generalised to any game that is built as a segment-based sequential process and features a built-in compressed representation of its game content.