mlsanityData to 14 Oct 2026

SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models

arXiv 2606.27741 ↗First posted 26 Jun 2026cs.CVcited by 0

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advances in video diffusion models have greatly improved visual fidelity, yet their generated motions often violate physical plausibility. We observe a common kinematic failure, "motion entanglement", the unintended coupling of independent motion sources, such as camera movement and object motion. We identify that this issue stems from data bias and the reconstruction-based training design of diffusion models. Training on noisy videos that still retain coarse motion cues inadvertently encourages the model to replicate existing motion without an incentive to learn how to model kinematically-grounded motions. To address this, we propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos rather than directly reconstructing real ones, breaking the reconstruction shortcut. We further employ motion-aware discriminative supervision and a progressive hard-case replay strategy to stabilize and accelerate learning. By leveraging freely-generated text prompts, our method can densely cover a broad motion space, including rare or finely-disentangled scenarios that would be costly to collect as video data. Extensive experiments demonstrate that our approach substantially improves the physical realism, motion disentanglement, and controllability of generated videos.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 65 of its 70 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

65 of 70 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation24 Sep
  2. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models29 May
  3. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise14 Jan
  4. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation30 Nov
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  6. 2024Training-free Camera Control for Video Generation14 Jun
  7. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
  8. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  9. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  10. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Models12 Dec
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  12. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning21 Nov
5 more
  1. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  2. 2023LLM-grounded Video Diffusion Models29 Sep
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 48

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Flow Matching for Generative Modelingcited by 5,702
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
36 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  3. 2018SlowFast Networks for Video Recognitioncited by 1,867
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  7. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  8. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  9. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  10. 2025Mean Flows for One-step Generative Modelingcited by 561
  11. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  12. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  13. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  14. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  15. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  16. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  17. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  18. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  19. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  20. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  21. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
  22. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  23. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  24. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  25. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 113
  26. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  27. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  28. 2025Towards Understanding Camera Motions in Any Videocited by 64
  29. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Modelcited by 63
  30. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 60
  31. 2025UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captionscited by 57
  32. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 53
  33. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  34. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  35. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 42
  36. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show
  2. 2026Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
  3. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusion
  4. 2026CausalMotion: Structured Physical Reasoning as Keyframe and Trajectory Guidance for Training-Free Video Generation
  5. 2026LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation
  6. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models
  7. 2026Seeking Physics in Diffusion Noise
  8. 2026DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generation