# Autoverse: An Evolvable Game Language for Learning Robust Embodied Agents

arXiv: https://arxiv.org/abs/2407.04221 · HTML: https://mlsanity.com/paper/2407.04221/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation](https://mlsanity.com/paper/1806.10729/) (2018)
- [Deep Learning for Procedural Content Generation](https://mlsanity.com/paper/2010.04548/) (2020)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning](https://mlsanity.com/paper/1902.01378/) (2019)
- [Learning to Generate Levels From Nothing](https://mlsanity.com/paper/2002.05259/) (2020)
- [Stabilizing Unsupervised Environment Design with a Learned Adversary](https://mlsanity.com/paper/2308.10797/) (2023)
- [Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning](https://mlsanity.com/paper/2311.03736/) (2023)
- [Pathfinding Neural Cellular Automata](https://mlsanity.com/paper/2301.06820/) (2023)

## Built on by (most cited first, 3 of 3)

- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [In-Context Reinforcement Learning under Non-Stationarity: A Survey](https://mlsanity.com/paper/2607.11906/) (2026)
- [The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning](https://mlsanity.com/paper/2608.26116/) (2026)

## Abstract (arXiv)

We introduce Autoverse, an evolvable, domain-specific language for single-player 2D grid-based games, and demonstrate its use as a scalable training ground for Open-Ended Learning (OEL) algorithms. Autoverse uses cellular-automaton-like rewrite rules to describe game mechanics, allowing it to express various game environments (e.g. mazes, dungeons, sokoban puzzles) that are popular testbeds for Reinforcement Learning (RL) agents. Each rewrite rule can be expressed as a series of simple convolutions, allowing for environments to be parallelized on the GPU, thereby drastically accelerating RL training. Using Autoverse, we propose jump-starting open-ended learning by imitation learning from search. In such an approach, we first evolve Autoverse environments (their rules and initial map topology) to maximize the number of iterations required by greedy tree search to discover a new best solution, producing a curriculum of increasingly complex environments and playtraces. We then distill these expert playtraces into a neural-network-based policy using imitation learning. Finally, we use the learned policy as a starting point for open-ended RL, where new training environments are continually evolved to maximize the RL player agent's value function error (a proxy for its regret, or the learnability of generated environments), finding that this approach improves the performance and generality of resultant player agents.