# Pixel VQ-VAEs for Improved Pixel Art Representation

arXiv: https://arxiv.org/abs/2203.12130 · HTML: https://mlsanity.com/paper/2203.12130/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 23 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Controllable Text-to-Image Generation](https://mlsanity.com/paper/1909.07083/) (2019)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [The VGLC: The Video Game Level Corpus](https://mlsanity.com/paper/1606.07487/) (2016)
- [Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach](https://mlsanity.com/paper/2008.04200/) (2020)
- [Tile Embedding: A General Representation for Procedural Level Generation via Machine Learning](https://mlsanity.com/paper/2110.03181/) (2021)
- [Autoencoding Generative Adversarial Networks](https://mlsanity.com/paper/2004.05472/) (2020)
- [Generating Gameplay-Relevant Art Assets with Transfer Learning](https://mlsanity.com/paper/2010.01681/) (2020)

## Built on by (most cited first, 4 of 4)

- [The Five-Dollar Model: Generating Game Maps and Sprites from Sentence Embeddings](https://mlsanity.com/paper/2308.04052/) (2023)
- [SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches](https://mlsanity.com/paper/2209.00185/) (2022)
- [Clustering-based Tile Embedding (CTE): A General Representation for Level Design with Skewed Tile Distributions](https://mlsanity.com/paper/2210.12789/) (2022)
- [A Missing Data Imputation GAN for Character Sprite Generation](https://mlsanity.com/paper/2409.10721/) (2024)

## Abstract (arXiv)

Machine learning has had a great deal of success in image processing. However, the focus of this work has largely been on realistic images, ignoring more niche art styles such as pixel art. Additionally, many traditional machine learning models that focus on groups of pixels do not work well with pixel art, where individual pixels are important. We propose the Pixel VQ-VAE, a specialized VQ-VAE model that learns representations of pixel art. We show that it outperforms other models in both the quality of embeddings as well as performance on downstream tasks.