# Breathing Life Into Sketches Using Text-to-Video Priors

arXiv: https://arxiv.org/abs/2311.13608 · HTML: https://mlsanity.com/paper/2311.13608/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 101 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 30 of 62)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Text-To-4D Dynamic Scene Generation](https://mlsanity.com/paper/2301.11280/) (2023)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)

## Built on by (most cited first, 30 of 48)

- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [ToonCrafter: Generative Cartoon Interpolation](https://mlsanity.com/paper/2405.17933/) (2024)
- [Sora as a World Model? A Complete Survey on Text-to-Video Generation](https://mlsanity.com/paper/2403.05131/) (2024)
- [PhysAnimator: Physics-Guided Generative Cartoon Animation](https://mlsanity.com/paper/2501.16550/) (2025)
- [Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation](https://mlsanity.com/paper/2408.09787/) (2024)
- [Text-to-Vector Generation with Neural Path Representation](https://mlsanity.com/paper/2405.10317/) (2024)
- [SketchAgent: Language-Driven Sequential Sketch Generation](https://mlsanity.com/paper/2411.17673/) (2024)
- [AniClipart: Clipart Animation with Text-to-Video Priors](https://mlsanity.com/paper/2404.12347/) (2024)
- [NeuralSVG: An Implicit Representation for Text-to-Vector Generation](https://mlsanity.com/paper/2501.03992/) (2025)
- [Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation](https://mlsanity.com/paper/2504.19189/) (2025)
- [Dynamic Typography: Bringing Text to Life via Video Diffusion Prior](https://mlsanity.com/paper/2404.11614/) (2024)
- [Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback](https://mlsanity.com/paper/2604.20730/) (2026)
- [MoVer: Motion Verification for Motion Graphics Animations](https://mlsanity.com/paper/2502.13372/) (2025)
- [VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control](https://mlsanity.com/paper/2502.01101/) (2025)
- [Multi-Object Sketch Animation with Grouping and Motion Trajectory Priors](https://mlsanity.com/paper/2508.15535/) (2025)
- [FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations](https://mlsanity.com/paper/2411.10818/) (2024)
- [DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance](https://mlsanity.com/paper/2512.10894/) (2025)
- [AutoMV: An Automatic Multi-Agent System for Music Video Generation](https://mlsanity.com/paper/2512.12196/) (2025)
- [Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints](https://mlsanity.com/paper/2411.19381/) (2024)
- [VAnim: Rendering-Aware Sparse State Modeling for Structure-Preserving Vector Animation](https://mlsanity.com/paper/2605.01517/) (2026)
- [LottieGPT: Tokenizing Vector Animation for Autoregressive Generation](https://mlsanity.com/paper/2604.11792/) (2026)
- [AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoising](https://mlsanity.com/paper/2402.03549/) (2024)
- [Multi-Object Sketch Animation by Scene Decomposition and Motion Planning](https://mlsanity.com/paper/2503.19351/) (2025)
- [WIR3D: Visually-Informed and Geometry-Aware 3D Shape Abstraction](https://mlsanity.com/paper/2505.04813/) (2025)
- [Neural Image Abstraction Using Long Smoothing B-Splines](https://mlsanity.com/paper/2511.05360/) (2025)
- [FlexiClip: Locality-Preserving Free-Form Character Animation](https://mlsanity.com/paper/2501.08676/) (2025)
- [OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens](https://mlsanity.com/paper/2603.02138/) (2026)
- [AirSketch: Generative Motion to Sketch](https://mlsanity.com/paper/2407.08906/) (2024)
- [AutoSketch: VLM-assisted Style-Aware Vector Sketch Completion](https://mlsanity.com/paper/2502.06860/) (2025)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)

## Abstract (arXiv)

A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually. An animated sketch opens another dimension to the expression of ideas and is widely used by designers for a variety of purposes. Animating sketches is a laborious process, requiring extensive experience and professional design skills. In this work, we present a method that automatically adds motion to a single-subject sketch (hence, "breathing life into it"), merely by providing a text prompt indicating the desired motion. The output is a short animation provided in vector representation, which can be easily edited. Our method does not require extensive training, but instead leverages the motion prior of a large pretrained text-to-video diffusion model using a score-distillation loss to guide the placement of strokes. To promote natural and smooth motion and to better preserve the sketch's appearance, we model the learned motion through two components. The first governs small local deformations and the second controls global affine transformations. Surprisingly, we find that even models that struggle to generate sketch videos on their own can still serve as a useful backbone for animating abstract representations.