# MarioNette: Self-Supervised Sprite Learning

arXiv: https://arxiv.org/abs/2104.14553 · HTML: https://mlsanity.com/paper/2104.14553/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 30 of 35)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing](https://mlsanity.com/paper/1803.01837/) (2018)
- [Learning to Simulate Dynamic Environments with GameGAN](https://mlsanity.com/paper/2005.12126/) (2020)
- [Layered Neural Rendering for Retiming People in Video](https://mlsanity.com/paper/2009.07833/) (2020)
- [Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking](https://mlsanity.com/paper/1911.09033/) (2019)
- [Investigating Human Priors for Playing Video Games](https://mlsanity.com/paper/1802.10217/) (2018)
- [Deep Learning for Video Game Playing](https://mlsanity.com/paper/1708.07902/) (2017)
- [Deep Parametric Shape Predictions using Distance Fields](https://mlsanity.com/paper/1904.08921/) (2019)
- [Partial Convolution based Padding](https://mlsanity.com/paper/1811.11718/) (2018)
- [Deep Generative Video Compression](https://mlsanity.com/paper/1810.02845/) (2018)

## Built on by (most cited first, 30 of 35)

- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [Deformable Sprites for Unsupervised Video Decomposition](https://mlsanity.com/paper/2204.07151/) (2022)
- [Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames](https://mlsanity.com/paper/2302.04973/) (2023)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Unsupervised Image Representation Learning with Deep Latent Particles](https://mlsanity.com/paper/2205.15821/) (2022)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles](https://mlsanity.com/paper/2306.05957/) (2023)
- [Boosting Object Representation Learning via Motion and Object Continuity](https://mlsanity.com/paper/2211.09771/) (2022)
- [Unlocking Slot Attention by Changing Optimal Transport Costs](https://mlsanity.com/paper/2301.13197/) (2023)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [Unsupervised Multi-object Segmentation Using Attention and Soft-argmax](https://mlsanity.com/paper/2205.13271/) (2022)
- [FactorMatte: Redefining Video Matting for Re-Composition Tasks](https://mlsanity.com/paper/2211.02145/) (2022)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [Neural Radiance Field Codebooks](https://mlsanity.com/paper/2301.04101/) (2023)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Differentiable Mathematical Programming for Object-Centric Representation Learning](https://mlsanity.com/paper/2210.02159/) (2022)
- [Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scans](https://mlsanity.com/paper/2304.09704/) (2023)
- [Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy](https://mlsanity.com/paper/2411.00868/) (2024)
- [The Learnable Typewriter: A Generative Approach to Text Analysis](https://mlsanity.com/paper/2302.01660/) (2023)
- [Search for Concepts: Discovering Visual Concepts Using Direct Optimization](https://mlsanity.com/paper/2210.14808/) (2022)
- [OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning](https://mlsanity.com/paper/2306.09682/) (2023)
- [Fast Sprite Decomposition from Animated Graphics](https://mlsanity.com/paper/2408.03923/) (2024)
- [Learning Explicit Object-Centric Representations with Vision Transformers](https://mlsanity.com/paper/2210.14139/) (2022)
- [Compositional Scene Modeling with Global Object-Centric Representations](https://mlsanity.com/paper/2211.11500/) (2022)
- [Towards Unsupervised Visual Reasoning: Do Off-The-Shelf Features Know How to Reason?](https://mlsanity.com/paper/2212.10292/) (2022)
- [Joint Level Generation and Translation Using Gameplay Videos](https://mlsanity.com/paper/2306.16662/) (2023)
- [HyperNVD: Accelerating Neural Video Decomposition via Hypernetworks](https://mlsanity.com/paper/2503.17276/) (2025)
- [Historical Printed Ornaments: Dataset and Tasks](https://mlsanity.com/paper/2408.08633/) (2024)

## Abstract (arXiv)

Artists and video game designers often construct 2D animations using libraries of sprites -- textured patches of objects and characters. We propose a deep learning approach that decomposes sprite-based video animations into a disentangled representation of recurring graphic elements in a self-supervised manner. By jointly learning a dictionary of possibly transparent patches and training a network that places them onto a canvas, we deconstruct sprite-based content into a sparse, consistent, and explicit representation that can be easily used in downstream tasks, like editing or analysis. Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision.