mlsanityData to 7 Oct 2026

MotionCraft: Physics-based Zero-Shot Video Generation

arXiv 2405.13557 ↗First posted 22 May 2024cs.LG cs.AI cs.CVcited by 46

On the map: Diffusion models → Video generation

Abstract (arXiv)

Generating videos with realistic and physically plausible motion is one of the main recent challenges in computer vision. While diffusion models are achieving compelling results in image generation, video diffusion models are limited by heavy training and huge models, resulting in videos that are still biased to the training dataset. In this work we propose MotionCraft, a new zero-shot video generator to craft physics-based and realistic videos. MotionCraft is able to warp the noise latent space of an image diffusion model, such as Stable Diffusion, by applying an optical flow derived from a physics simulation. We show that warping the noise latent space results in coherent application of the desired motion while allowing the model to generate missing elements consistent with the scene evolution, which would otherwise result in artefacts or missing content if the flow was applied in the pixel space. We compare our method with the state-of-the-art Text2Video-Zero reporting qualitative and quantitative improvements, demonstrating the effectiveness of our approach to generate videos with finely-prescribed complex motion dynamics. Project page: https://mezzelfo.github.io/MotionCraft/

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 29 of its 37 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

29 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models3 Dec
  2. 2023LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation1 Nov
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  4. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Models24 Mar
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  6. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  9. 2022Video Diffusion Models7 Apr

Other references · 20

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
8 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  5. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  6. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  7. 2020Learning to Control PDEs with Differentiable Physicscited by 131
  8. 2024Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimatorscited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 18.

46 papers.

Sort:

By area:

  1. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  2. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  3. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  4. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  5. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  6. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  7. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  8. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  9. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  10. 2025Generative Physical AI in Vision: A Surveycited by 24
  11. 2025A Survey of Interactive Generative Videocited by 23
  12. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
34 more
  1. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  2. 2026Self-Refining Video Samplingcited by 14
  3. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  4. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Modelscited by 13
  5. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 12
  6. 2026Physical Simulator In-the-Loop Video Generationcited by 12
  7. 2025Grounding Creativity in Physics: A Brief Survey of Physical Priors in AIGCcited by 11
  8. 2025ProPhy: Progressive Physical Alignment for Dynamic World Simulationcited by 11
  9. 2025CAD-VAE: Leveraging Correlation-Aware Latents for Comprehensive Fair Disentanglementcited by 10
  10. 2025PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learningcited by 10
  11. 2026Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamicscited by 10
  12. 2026Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goalscited by 9
  13. 2025Towards Physical Understanding in Video Generation: A 3D Point Regularization Approachcited by 8
  14. 2025Product of Experts for Visual Generationcited by 6
  15. 2025Planning with Sketch-Guided Verification for Physics-Aware Video Generationcited by 6
  16. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  17. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  18. 2026MoRight: Motion Control Done Rightcited by 2
  19. 2026PhiZero: A World Model Built Around Physical Languagecited by 2
  20. 2025Physics-Guided Motion Loss for Video Generation Modelcited by 1
  21. 2025Can Image-To-Video Models Simulate Pedestrian Dynamics?cited by 1
  22. 2026Physics-Informed Video Diffusion For Shallow Water Equationscited by 1
  23. 2026Tempered Self-Similarity Alignment for Physically Plausible Video Generationcited by 1
  24. 2026NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamicscited by 1
  25. 2026Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generationcited by 1
  26. 2026Can Video World Models Track Unobserved World States?cited by 1
  27. 2025PEDRA: Evaluating the Realism of Pedestrian Dynamics in Video Generationcited by 0
  28. 2026MechVerse: Evaluating Physical Motion Consistency in Video Generation Modelscited by 0
  29. 2026CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENerationcited by 0
  30. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  31. 2026Learning Implicit Constitutive Laws for Dynamic 3D Gaussian Splatting from Monocular Videoscited by 0
  32. 2026VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinementcited by 0
  33. 2026Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanismscited by 0
  34. 2026Reimagine Video Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Noise Crystallization and Liquid Noise: Zero-shot Video Generation using Image Diffusion Models
  2. 2024Controllable Longer Image Animation with Diffusion Models
  3. 2024Fine-gained Zero-shot Video Sampling
  4. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation
  5. 2024ViMo: Generating Motions from Casual Videos
  6. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  7. 2025RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realism
  8. 2023TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models