# SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches

arXiv: https://arxiv.org/abs/2209.00185 · HTML: https://mlsanity.com/paper/2209.00185/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [Latent Video Transformer](https://mlsanity.com/paper/2006.10704/) (2020)
- [World-Consistent Video-to-Video Synthesis](https://mlsanity.com/paper/2007.08509/) (2020)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)
- [Deep Animation Video Interpolation in the Wild](https://mlsanity.com/paper/2104.02495/) (2021)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [Pose-Guided Human Animation from a Single Image in the Wild](https://mlsanity.com/paper/2012.03796/) (2020)
- [Generating Gameplay-Relevant Art Assets with Transfer Learning](https://mlsanity.com/paper/2010.01681/) (2020)
- [Pixel VQ-VAEs for Improved Pixel Art Representation](https://mlsanity.com/paper/2203.12130/) (2022)
- [Video Frame Interpolation via Structure-Motion based Iterative Fusion](https://mlsanity.com/paper/2105.05353/) (2021)

## Built on by (most cited first, 6 of 6)

- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [LinkTo-Anime: A 2D Animation Optical Flow Dataset from 3D Model Rendering](https://mlsanity.com/paper/2506.02733/) (2025)
- [DACoN: DINO for Anime Paint Bucket Colorization with Any Number of Reference Images](https://mlsanity.com/paper/2509.14685/) (2025)
- [A Missing Data Imputation GAN for Character Sprite Generation](https://mlsanity.com/paper/2409.10721/) (2024)
- [SketchKeyAnime: Reference-anchored Sparse Key-Sketch Animation Synthesis](https://mlsanity.com/paper/2606.19958/) (2026)

## Abstract (arXiv)

2D animation is a common factor in game development, used for characters, effects and background art. It involves work that takes both skill and time, but parts of which are repetitive and tedious. Automated animation approaches exist, but are designed without animators in mind. The focus is heavily on real-life video, which follows strict laws of how objects move, and does not account for the stylistic movement often present in 2D animation. We propose a problem formulation that more closely adheres to the standard workflow of animation. We also demonstrate a model, SketchBetween, which learns to map between keyframes and sketched in-betweens to rendered sprite animations. We demonstrate that our problem formulation provides the required information for the task and that our model outperforms an existing method.