# Joint Level Generation and Translation Using Gameplay Videos

arXiv: https://arxiv.org/abs/2306.16662 · HTML: https://mlsanity.com/paper/2306.16662/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 17 of 17)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [The VGLC: The Video Game Level Corpus](https://mlsanity.com/paper/1606.07487/) (2016)
- [MarioNette: Self-Supervised Sprite Learning](https://mlsanity.com/paper/2104.14553/) (2021)
- [Generating and Blending Game Levels via Quality-Diversity in the Latent Space of a Variational Autoencoder](https://mlsanity.com/paper/2102.12463/) (2021)
- [Controllable Level Blending between Games using Variational Autoencoders](https://mlsanity.com/paper/2002.11869/) (2020)
- [Exploring Level Blending across Platformers via Paths and Affordances](https://mlsanity.com/paper/2009.06356/) (2020)
- [Dungeon and Platformer Level Blending and Generation using Conditional VAEs](https://mlsanity.com/paper/2106.12692/) (2021)
- [Tile Embedding: A General Representation for Procedural Level Generation via Machine Learning](https://mlsanity.com/paper/2110.03181/) (2021)
- [Entity Embedding as Game Representation](https://mlsanity.com/paper/2010.01685/) (2020)

## Built on by (most cited first, 1 of 1)

- [Representing and Generating Levels Over Time through Playtrace Reconstructive Partitioning](https://mlsanity.com/paper/2607.12097/) (2026)

## Abstract (arXiv)

Procedural Content Generation via Machine Learning (PCGML) faces a significant hurdle that sets it apart from other fields, such as image or text generation, which is limited annotated data. Many existing methods for procedural level generation via machine learning require a secondary representation besides level images. However, the current methods for obtaining such representations are laborious and time-consuming, which contributes to this problem. In this work, we aim to address this problem by utilizing gameplay videos of two human-annotated games to develop a novel multi-tail framework that learns to perform simultaneous level translation and generation. The translation tail of our framework can convert gameplay video frames to an equivalent secondary representation, while its generation tail can produce novel level segments. Evaluation results and comparisons between our framework and baselines suggest that combining the level generation and translation tasks can lead to an overall improved performance regarding both tasks. This represents a possible solution to limited annotated level data, and we demonstrate the potential for future versions to generalize to unseen games.