mlsanityData to 7 Oct 2026

Breathing Life Into Sketches Using Text-to-Video Priors

arXiv 2311.13608 ↗First posted 21 Nov 2023cs.CV cs.GR cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually. An animated sketch opens another dimension to the expression of ideas and is widely used by designers for a variety of purposes. Animating sketches is a laborious process, requiring extensive experience and professional design skills. In this work, we present a method that automatically adds motion to a single-subject sketch (hence, "breathing life into it"), merely by providing a text prompt indicating the desired motion. The output is a short animation provided in vector representation, which can be easily edited. Our method does not require extensive training, but instead leverages the motion prior of a large pretrained text-to-video diffusion model using a score-distillation loss to guide the placement of strokes. To promote natural and smooth motion and to better preserve the sketch's appearance, we model the learned motion through two components. The first governs small local deformations and the second controls global affine transformations. Surprisingly, we find that even models that struggle to generate sketch videos on their own can still serve as a useful backbone for animating abstract representations.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 62 of its 101 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

62 of 101 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  10. 2023ModelScope Text-to-Video Technical Reportcited by 651
  11. 2015Holistically-Nested Edge Detectioncited by 590
  12. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
38 more
  1. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  2. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  3. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  4. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  5. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  6. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  7. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  8. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  9. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  10. 2017A Neural Representation of Sketch Drawingscited by 311
  11. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  12. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  13. 2018Stochastic Video Generation with a Learned Priorcited by 244
  14. 2017Stochastic Variational Video Predictioncited by 234
  15. 2023Text-To-4D Dynamic Scene Generationcited by 204
  16. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  17. 2019Scaling Autoregressive Video Modelscited by 189
  18. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  19. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  20. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  21. 2017Video Generation From Textcited by 153
  22. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  23. 2022VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Modelscited by 112
  24. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  25. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  26. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  27. 2022Learning to generate line drawings that convey geometry and semanticscited by 82
  28. 2020Sketchformer: Transformer-based Representation for Sketched Structurecited by 78
  29. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generationcited by 74
  30. 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 67
  31. 2022CLIPascene: Scene Sketching with Different Types and Levels of Abstractioncited by 63
  32. 2023Word-As-Image for Semantic Typographycited by 62
  33. 2018Photo Wake-Up: 3D Character Animation from a Single Photocited by 60
  34. 2020Deep Learning for Free-Hand Sketch: A Surveycited by 59
  35. 2018Learning to Sketch with Shortcut Cycle Consistencycited by 47
  36. 2020Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestaltcited by 42
  37. 2018Learning Deep Sketch Abstractioncited by 36
  38. 2020Why Do Line Drawings Work? A Realism Hypothesiscited by 28

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 22; 2026: 11.

47 papers.

Sort:

By area:

  1. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  2. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  3. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  4. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  5. 2024Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generationcited by 35
  6. 2024Text-to-Vector Generation with Neural Path Representationcited by 32
  7. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  8. 2024AniClipart: Clipart Animation with Text-to-Video Priorscited by 17
  9. 2025NeuralSVG: An Implicit Representation for Text-to-Vector Generationcited by 16
  10. 2025Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animationcited by 16
  11. 2024Dynamic Typography: Bringing Text to Life via Video Diffusion Priorcited by 12
  12. 2026Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedbackcited by 12
35 more
  1. 2025MoVer: Motion Verification for Motion Graphics Animationscited by 10
  2. 2025VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Controlcited by 9
  3. 2025Multi-Object Sketch Animation with Grouping and Motion Trajectory Priorscited by 9
  4. 2024FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animationscited by 8
  5. 2024Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraintscited by 7
  6. 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 7
  7. 2025DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidancecited by 6
  8. 2026VAnim: Rendering-Aware Sparse State Modeling for Structure-Preserving Vector Animationcited by 6
  9. 2026LottieGPT: Tokenizing Vector Animation for Autoregressive Generationcited by 5
  10. 2024AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoisingcited by 4
  11. 2025Multi-Object Sketch Animation by Scene Decomposition and Motion Planningcited by 4
  12. 2025FlexiClip: Locality-Preserving Free-Form Character Animationcited by 3
  13. 2025WIR3D: Visually-Informed and Geometry-Aware 3D Shape Abstractioncited by 3
  14. 2025Neural Image Abstraction Using Long Smoothing B-Splinescited by 3
  15. 2024AirSketch: Generative Motion to Sketchcited by 2
  16. 2025AutoSketch: VLM-assisted Style-Aware Vector Sketch Completioncited by 2
  17. 2025From Fragment to One Piece: A Survey on AI-Driven Graphic Designcited by 2
  18. 2025Recovering Dynamic 3D Sketches from Videoscited by 2
  19. 2025Style Customization of Text-to-Vector Generation with Image Diffusion Priorscited by 2
  20. 2025LINR Bridge: Vector Graphic Animation via Neural Implicits and Video Diffusion Priorscited by 2
  21. 2026VideoSketcher: Sequential Sketch Generation Using Video Model Priorscited by 2
  22. 2026OmniLottie: Generating Vector Animations via Parameterized Lottie Tokenscited by 2
  23. 2024AMG: Avatar Motion Guided Video Generationcited by 1
  24. 2025CasTex: Cascaded Text-to-Texture Synthesis via Explicit Texture Maps and Physically-Based Shadingcited by 1
  25. 2025Birth of a Painting: Differentiable Brushstroke Reconstructioncited by 1
  26. 2026Self-Consistency for LLM-Based Motion Trajectory Generation and Verificationcited by 1
  27. 2026Sketch2Motion: Text-driven 2D Sketch to 3D Animation via Diffusion-guided Skeleton Optimizationcited by 1
  28. 2026LiveSVG: Zero-Shot SVG Animation via Video Generationcited by 1
  29. 2024Towards motion from video diffusion modelscited by 0
  30. 2025SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Modelscited by 0
  31. 2025Vector sketch animation generation with differentiable motion trajectoriescited by 0
  32. 20254-Doodle: Text to 3D Sketches that Move!cited by 0
  33. 2026ProxyImg: Towards Highly-Controllable Image Representation via Hierarchical Disentangled Proxy Embeddingcited by 0
  34. 2026Single-Line Drawing Generation via Semantics-Driven Optimizationcited by 0
  35. 2026Strike a Chord! Modal Kinetic Typographycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SketchKeyAnime: Reference-anchored Sparse Key-Sketch Animation Synthesis
  2. 2026SketchDynamics: Exploring Free-Form Sketches for Dynamic Intent Expression in Animation Generation ↗
  3. 2023SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation
  4. 2025AnimaMimic: Imitating 3D Animation from Video Priors
  5. 2023AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
  6. 2023Enhanced Fine-grained Motion Diffusion for Text-driven Human Motion Synthesis
  7. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  8. 2025SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation