# Sketch2Motion: Text-driven 2D Sketch to 3D Animation via Diffusion-guided Skeleton Optimization

arXiv: https://arxiv.org/abs/2605.28394 · HTML: https://mlsanity.com/paper/2605.28394/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assets](https://mlsanity.com/paper/2505.07747/) (2025)
- [TC4D: Trajectory-Conditioned Text-to-4D Generation](https://mlsanity.com/paper/2403.17920/) (2024)
- [Photo Wake-Up: 3D Character Animation from a Single Photo](https://mlsanity.com/paper/1812.02246/) (2018)
- [Breathing Life Into Sketches Using Text-to-Video Priors](https://mlsanity.com/paper/2311.13608/) (2023)
- [Puppeteer: Rig and Animate Your 3D Models](https://mlsanity.com/paper/2508.10898/) (2025)
- [AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animation](https://mlsanity.com/paper/2506.09982/) (2025)
- [A Method for Animating Children's Drawings of the Human Figure](https://mlsanity.com/paper/2303.12741/) (2023)
- [AniClipart: Clipart Animation with Text-to-Video Priors](https://mlsanity.com/paper/2404.12347/) (2024)
- [Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation](https://mlsanity.com/paper/2504.19189/) (2025)
- [AnimaX: Animating the Inanimate in 3D with Joint Video-Pose Diffusion Models](https://mlsanity.com/paper/2506.19851/) (2025)
- [SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control](https://mlsanity.com/paper/2512.03028/) (2025)
- [Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis](https://mlsanity.com/paper/2601.14253/) (2026)
- [Articulated Kinematics Distillation from Video Diffusion Models](https://mlsanity.com/paper/2504.01204/) (2025)
- [Dynamic Typography: Bringing Text to Life via Video Diffusion Prior](https://mlsanity.com/paper/2404.11614/) (2024)
- [DrawingSpinUp: 3D Animation from Single Character Drawings](https://mlsanity.com/paper/2409.08615/) (2024)
- [Multi-Object Sketch Animation with Grouping and Motion Trajectory Priors](https://mlsanity.com/paper/2508.15535/) (2025)
- [FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations](https://mlsanity.com/paper/2411.10818/) (2024)
- [Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints](https://mlsanity.com/paper/2411.19381/) (2024)
- [AnimaMimic: Imitating 3D Animation from Video Priors](https://mlsanity.com/paper/2512.14133/) (2025)
- [Bringing Objects to Life: training-free 4D generation from 3D objects through view consistent noise](https://mlsanity.com/paper/2412.20422/) (2024)
- [Animus3D: Text-driven 3D Animation via Motion Score Distillation](https://mlsanity.com/paper/2512.12534/) (2025)
- [RigMo: Unifying Rig and Motion Learning for Generative Animation](https://mlsanity.com/paper/2601.06378/) (2026)
- [Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging](https://mlsanity.com/paper/2602.04805/) (2026)

## Built on by (most cited first, 1 of 1)

- [Strike a Chord! Modal Kinetic Typography](https://mlsanity.com/paper/2609.38325/) (2026)

## Abstract (arXiv)

Animation of 2D hand-drawn sketches provides an effective medium for visual communication. However, these sketches pose challenges, particularly in handling occlusions and accurately mapping motion. While 3D animation naturally addresses these challenges, estimating 3D motion remains a very complex task. Recent approaches to converting 2D sketches to 3D animations have mainly focused on specific types of motion, such as bipedal movements and facial expressions. We propose Sketch2Motion, a diffusion-guided framework for skeleton-based motion synthesis that combines classical character animation pipelines with deep generative priors. Our method represents motion using skeletal transformations, which are propagated to mesh deformations via linear blend skinning. To guide the resulting animation toward realistic and semantically meaningful motion, we integrate a text-to-video diffusion model via motion-aware score-distillation sampling (MoSDS), enabling optimization without paired motion data. Additionally, we apply physics-inspired smoothness, topological, and contact constraints to stabilize optimization and preserve motion plausibility. Further, we integrate a spring-mass simulator to introduce secondary motion effects. The proposed framework is generalized, fully differentiable, modular, and compatible with biped, quadruped, and non-living articulated characters. Experiments demonstrate that our approach produces temporally coherent, text-aligned animations that outperform baseline motion transfer methods that lack generative priors or explicit physical constraints. We will make our code and dataset publicly available.