# Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics

arXiv: https://arxiv.org/abs/2408.04631 · HTML: https://mlsanity.com/paper/2408.04631/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 84 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Builds on (most cited first, 30 of 74)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)

## Built on by (most cited first, 28 of 28)

- [MotionStream: Real-Time Video Generation with Interactive Motion Controls](https://mlsanity.com/paper/2511.01266/) (2025)
- [Anymate: A Dataset and Baselines for Learning 3D Object Rigging](https://mlsanity.com/paper/2505.06227/) (2025)
- [DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness](https://mlsanity.com/paper/2503.22677/) (2025)
- [DreamArt: Generating Interactable Articulated Objects from a Single Image](https://mlsanity.com/paper/2507.05763/) (2025)
- [Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention](https://mlsanity.com/paper/2410.10774/) (2024)
- [Particulate: Feed-Forward 3D Object Articulation](https://mlsanity.com/paper/2512.11798/) (2025)
- [PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model](https://mlsanity.com/paper/2503.19913/) (2025)
- [Motion Modes: What Could Happen Next?](https://mlsanity.com/paper/2412.00148/) (2024)
- [TACO: Taming Diffusion for in-the-wild Video Amodal Completion](https://mlsanity.com/paper/2503.12049/) (2025)
- [DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors](https://mlsanity.com/paper/2409.08278/) (2024)
- [Articulate That Object Part (ATOP): 3D Part Articulation via Text and Motion Personalization](https://mlsanity.com/paper/2502.07278/) (2025)
- [PoseTraj: Pose-Aware Trajectory Control in Video Diffusion](https://mlsanity.com/paper/2503.16068/) (2025)
- [In-2-4D: Inbetweening from Two Single-View Images to 4D Generation](https://mlsanity.com/paper/2504.08366/) (2025)
- [MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstruction](https://mlsanity.com/paper/2603.19231/) (2026)
- [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025)
- [ShapeUP: Scalable Image-Conditioned 3D Editing](https://mlsanity.com/paper/2602.05676/) (2026)
- [BiMotion: B-spline Motion for Text-guided Dynamic 3D Character Generation](https://mlsanity.com/paper/2602.18873/) (2026)
- [Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control](https://mlsanity.com/paper/2606.14699/) (2026)
- [What If : Understanding Motion Through Sparse Interactions](https://mlsanity.com/paper/2510.12777/) (2025)
- [AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation](https://mlsanity.com/paper/2603.25726/) (2026)
- [Envisioning the Future, One Step at a Time](https://mlsanity.com/paper/2604.09527/) (2026)
- [Inferring Compositional 4D Scenes without Ever Seeing One](https://mlsanity.com/paper/2512.05272/) (2025)
- [Motion Marionette: Rethinking Rigid Motion Transfer via Prior Guidance](https://mlsanity.com/paper/2511.19909/) (2025)
- [A Study of the Design Space of Motion and Disocclusion Control in Video Generation](https://mlsanity.com/paper/2512.13392/) (2025)
- [PhysSkin: Real-Time and Generalizable Physics-Based Animation via Self-Supervised Neural Skinning](https://mlsanity.com/paper/2603.23194/) (2026)
- [DailyArt: Discovering Articulation from Single Static Images via Latent Dynamics](https://mlsanity.com/paper/2604.07758/) (2026)
- [PWM-ArtGen: Part World Model for Articulated Object Generation](https://mlsanity.com/paper/2607.02045/) (2026)
- [ArtiMo: Agent-Driven Articulated Mesh Animation](https://mlsanity.com/paper/2608.20699/) (2026)

## Abstract (arXiv)

We introduce Puppet-Master, an interactive video generator that captures the internal, part-level motion of objects, serving as a proxy for modeling object dynamics universally. Given an image of an object and a set of "drags" specifying the trajectory of a few points on the object, the model synthesizes a video where the object's parts move accordingly. To build Puppet-Master, we extend a pre-trained image-to-video generator to encode the input drags. We also propose all-to-first attention, an alternative to conventional spatial attention that mitigates artifacts caused by fine-tuning a video generator on out-of-domain data. The model is fine-tuned on Objaverse-Animation-HQ, a new dataset of curated part-level motion clips obtained by rendering synthetic 3D animations. Unlike real videos, these synthetic clips avoid confounding part-level motion with overall object and camera motion. We extensively filter sub-optimal animations and augment the synthetic renderings with meaningful drags that emphasize the internal dynamics of objects. We demonstrate that Puppet-Master learns to generate part-level motions, unlike other motion-conditioned video generators that primarily move the object as a whole. Moreover, Puppet-Master generalizes well to out-of-domain real images, outperforming existing methods on real-world benchmarks in a zero-shot manner.