# Sketch Video Synthesis

arXiv: https://arxiv.org/abs/2311.15306 · HTML: https://mlsanity.com/paper/2311.15306/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 26 Nov 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation](https://mlsanity.com/paper/2007.11301/) (2020)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [Blind Video Temporal Consistency via Deep Video Prior](https://mlsanity.com/paper/2010.11838/) (2020)
- [IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers](https://mlsanity.com/paper/2304.14400/) (2023)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)
- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)

## Built on by (most cited first, 4 of 4)

- [FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations](https://mlsanity.com/paper/2411.10818/) (2024)
- [Recovering Dynamic 3D Sketches from Videos](https://mlsanity.com/paper/2503.20321/) (2025)
- [Vector sketch animation generation with differentiable motion trajectories](https://mlsanity.com/paper/2509.25857/) (2025)
- [DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches](https://mlsanity.com/paper/2605.23508/) (2026)

## Abstract (arXiv)

Understanding semantic intricacies and high-level concepts is essential in image sketch generation, and this challenge becomes even more formidable when applied to the domain of videos. To address this, we propose a novel optimization-based framework for sketching videos represented by the frame-wise B'ezier curve. In detail, we first propose a cross-frame stroke initialization approach to warm up the location and the width of each curve. Then, we optimize the locations of these curves by utilizing a semantic loss based on CLIP features and a newly designed consistency loss using the self-decomposed 2D atlas network. Built upon these design elements, the resulting sketch video showcases impressive visual abstraction and temporal coherence. Furthermore, by transforming a video into SVG lines through the sketching process, our method unlocks applications in sketch-based video editing and video doodling, enabled through video composition, as exemplified in the teaser.