mlsanityData to 14 Oct 2026

VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior

arXiv 2503.23368 ↗First posted 30 Mar 2025cs.CV cs.AIcited by 52

On the map: Diffusion models → Video generation

Abstract (arXiv)

Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos and drawing the attention of the community in their potential as world simulators. However, despite their capabilities, VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics, resulting in incorrect dynamics and event sequences. To address this limitation, we propose a novel two-stage image-to-video generation framework that explicitly incorporates physics with vision and language informed physical prior. In the first stage, we employ a Vision Language Model (VLM) as a coarse-grained motion planner, integrating chain-of-thought and physics-aware reasoning to predict a rough motion trajectories/changes that approximate real-world physical dynamics while ensuring the inter-frame consistency. In the second stage, we use the predicted motion trajectories/changes to guide the video generation of a VDM. As the predicted motion trajectories/changes are rough, noise is added during inference to provide freedom to the VDM in generating motion with more fine details. Extensive experimental results demonstrate that our framework can produce physically plausible motion, and comparative evaluations highlight the notable superiority of our approach over existing methods. More video results are available on our Project Page: https://madaoer.github.io/projects/physically_plausible_video_generation.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 48 of its 71 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

48 of 71 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise14 Jan
  2. 2024LTX-Video: Realtime Video Latent Diffusion30 Dec
  3. 2024AKiRa: Augmentation Kit on Rays for optical video generation18 Dec
  4. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation30 Nov
  5. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers27 Nov
  6. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  7. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models24 Jun
  8. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model30 May
  9. 2024ReVideo: Remake a Video with Motion and Content Control22 May
  10. 2024MotionCraft: Physics-based Zero-Shot Video Generation22 May
  11. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
  12. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation31 Dec
3 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2023LLM-grounded Video Diffusion Models29 Sep
  3. 2022PhysDiff: Physics-Guided Human Motion Diffusion Model5 Dec

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  8. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  10. 2023Learning Interactive Real-World Simulatorscited by 458
  11. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  12. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
21 more
  1. 2024Navigation World Modelscited by 272
  2. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  3. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  4. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  5. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  6. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  7. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
  8. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  9. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  10. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  11. 2023Self-correcting LLM-controlled Diffusion Modelscited by 113
  12. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  13. 2023Generative Image Dynamicscited by 94
  14. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 93
  15. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  16. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  17. 2024Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimatorscited by 39
  18. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  19. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  20. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  21. 2024Bora: Biomedical Generalist Video Generation Modelcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 39.

52 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  2. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  3. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  4. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 37
  5. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 25
  6. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  7. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 19
  8. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  9. 2026Self-Refining Video Samplingcited by 15
  10. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  11. 2025VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learningcited by 12
  12. 2026Programmable World Modelcited by 10
38 more
  1. 2026Evolution of Video Generative Foundationscited by 7
  2. 2025Planning with Sketch-Guided Verification for Physics-Aware Video Generationcited by 6
  3. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  4. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  5. 2026PhysVid: Physics Aware Local Conditioning for Generative Video Modelscited by 6
  6. 2026Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generationcited by 6
  7. 2025STANCE: Motion Coherent Video Generation Via Sparse-to-Dense Anchored Encodingcited by 5
  8. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 5
  9. 2026PhyCo: Learning Controllable Physical Priors for Generative Motioncited by 5
  10. 2026Vision Language Models Cannot Reason About Physical Transformationcited by 4
  11. 2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 4
  12. 2026PhysOmni: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interactioncited by 4
  13. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  14. 2026DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generationcited by 2
  15. 2026MoRight: Motion Control Done Rightcited by 2
  16. 2026OptiWorld: Optimal Control for Video World Generation under Physical Constraintscited by 2
  17. 2026Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generationcited by 2
  18. 2026PhiZero: A World Model Built Around Physical Languagecited by 2
  19. 2026PhysWeep: Does a Video Generator Realize the Physics You Ask For?cited by 2
  20. 2026Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanismscited by 2
  21. 2026Seeking Physics in Diffusion Noisecited by 1
  22. 2026Physics-Informed Video Generation via Mixture-of-Experts Latent Alignmentcited by 1
  23. 2026NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamicscited by 1
  24. 2026DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Maskingcited by 1
  25. 2026muSync-GS: Physics-Synchronized Driving Video Synthesis for Weather and Geometric Road Hazardscited by 1
  26. 2026Physics-Aware Video Generation via Agentic Planning and Graph-Guided Optimizationcited by 1
  27. 2026Beyond Rigid: Benchmarking Non-Rigid Video Editingcited by 0
  28. 20263DPhysVideo: Consistency-Guided Flow SDE for Video Generation via 3D Scene Reconstruction and Physical Simulationcited by 0
  29. 2026PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generationcited by 0
  30. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  31. 2026SAGA: Stable Acceleration Guidance for Autoregressive Video Generationcited by 0
  32. 2026MirrorWorld: Taming Video Diffusion Models for Mirror Reflection Generationcited by 0
  33. 2026Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Modelscited by 0
  34. 2026Physically Plausible Video Generation via Visual-Semantic Chain-of-Events Conditioningcited by 0
  35. 2026VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinementcited by 0
  36. 2026Reimagine Video Dynamicscited by 0
  37. 2026No Corners Cut: State-Grounded Transitions for Mid-Stream Prompt Switches in Video Generationcited by 0
  38. 2026HiPhy: Hierarchical Alignment for Physically-Plausible Multi-Principle Video Generationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CausalMotion: Structured Physical Reasoning as Keyframe and Trajectory Guidance for Training-Free Video Generation
  2. 2025From Seeing to Predicting: A Vision-Language Framework for Trajectory Forecasting and Controlled Video Generation
  3. 2026PhyRPR: Training-Free Physics-Constrained Video Generation
  4. 2025Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement
  5. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusion
  6. 2025MoReGen: Multi-Agent Motion-Reasoning Engine for Code-based Text-to-Video Synthesis
  7. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility
  8. 2026MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modeling