mlsanityData to 14 Oct 2026

MarioNette: Self-Supervised Sprite Learning

arXiv 2104.14553 ↗First posted 29 Apr 2021cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Artists and video game designers often construct 2D animations using libraries of sprites -- textured patches of objects and characters. We propose a deep learning approach that decomposes sprite-based video animations into a disentangled representation of recurring graphic elements in a self-supervised manner. By jointly learning a dictionary of possibly transparent patches and training a network that places them onto a canvas, we deconstruct sprite-based content into a sparse, consistent, and explicit representation that can be easily used in downstream tasks, like editing or analysis. Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

35 of 58 references.

References · 35

  1. 2016Layer Normalizationcited by 5,669
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  4. 2015Fast R-CNNcited by 2,933
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  6. 2018Group Normalizationcited by 1,524
  7. 2015Spatial Transformer Networkscited by 1,429
  8. 2017Deep Image Priorcited by 1,098
  9. 2020Object-Centric Learning with Slot Attentioncited by 876
  10. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  11. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  12. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
23 more
  1. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  2. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 332
  3. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  4. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  5. 2019Seeing What a GAN Cannot Generatecited by 148
  6. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 137
  7. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  8. 2018"Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priorscited by 114
  9. 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
  10. 2018ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositingcited by 90
  11. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  12. 2020Layered Neural Rendering for Retiming People in Videocited by 51
  13. 2019Exploiting Spatial Invariance for Scalable Unsupervised Object Trackingcited by 40
  14. 2018Investigating Human Priors for Playing Video Gamescited by 39
  15. 2017Deep Learning for Video Game Playingcited by 30
  16. 2019Deep Parametric Shape Predictions using Distance Fieldscited by 30
  17. 2018Partial Convolution based Paddingcited by 29
  18. 2018Deep Generative Video Compressioncited by 23
  19. 2018Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackerscited by 22
  20. 2015Efficient inference in occlusion-aware generative models of imagescited by 11
  21. 2020Discovering Pattern Structure Using Differentiable Compositingcited by 11
  22. 2018Unsupervised Image Decomposition in Vector Layerscited by 5
  23. 2019StampNet: unsupervised multi-class object discoverycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 13; 2023: 9; 2024: 5; 2025: 2; 2026: 4.

35 papers.

Sort:

By area:

  1. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  2. 2022Deformable Sprites for Unsupervised Video Decompositioncited by 61
  3. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  4. 2024Generative Omnimatte: Learning to Decompose Video into Layerscited by 38
  5. 2022Compositional Scene Representation Learning via Reconstruction: A Surveycited by 27
  6. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  7. 2022Unsupervised Multi-object Segmentation by Predicting Probable Motion Patternscited by 19
  8. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 19
  9. 2022Boosting Object Representation Learning via Motion and Object Continuitycited by 17
  10. 2023Unlocking Slot Attention by Changing Optimal Transport Costscited by 16
  11. 2023Multi-Object Discovery by Low-Dimensional Object Motioncited by 16
  12. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
23 more
  1. 2022Unsupervised Multi-object Segmentation Using Attention and Soft-argmaxcited by 11
  2. 2022FactorMatte: Redefining Video Matting for Re-Composition Taskscited by 8
  3. 2021Layered Controllable Video Generationcited by 7
  4. 2023Neural Radiance Field Codebookscited by 7
  5. 2022Controllable Video Generation through Global and Local Motion Dynamicscited by 6
  6. 2022Differentiable Mathematical Programming for Object-Centric Representation Learningcited by 6
  7. 2023Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scanscited by 5
  8. 2024Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomycited by 4
  9. 2023The Learnable Typewriter: A Generative Approach to Text Analysiscited by 3
  10. 2022Search for Concepts: Discovering Visual Concepts Using Direct Optimizationcited by 2
  11. 2023OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learningcited by 2
  12. 2024Fast Sprite Decomposition from Animated Graphicscited by 2
  13. 2022Learning Explicit Object-Centric Representations with Vision Transformerscited by 1
  14. 2022Compositional Scene Modeling with Global Object-Centric Representationscited by 1
  15. 2022Towards Unsupervised Visual Reasoning: Do Off-The-Shelf Features Know How to Reason?cited by 1
  16. 2023Joint Level Generation and Translation Using Gameplay Videoscited by 1
  17. 2025HyperNVD: Accelerating Neural Video Decomposition via Hypernetworkscited by 1
  18. 2024Historical Printed Ornaments: Dataset and Taskscited by 0
  19. 2024Next state prediction gives rise to entangled, yet compositional representations of objectscited by 0
  20. 2025Finite Automata Extraction: Low-data World Model Learning as Programs from Gameplay Videocited by 0
  21. 2026Deep Sprite-based Image Models: An Analysiscited by 0
  22. 2026Leveraging Morphology for Historical Script Metrological Analysiscited by 0
  23. 2026Bottom-up Modeling of Repeated Elements via Single Image Analysis-by-Synthesiscited by 0