Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising
On the map: Diffusion models → Video generation
Abstract (arXiv)
While modern video diffusion models excel in visual fidelity, maintaining long-range physical consistency remains a formidable challenge. Conventional pixel-reconstruction objectives mainly focus on appearance details and often fail to capture the underlying dynamics of a scene. To mitigate this, recent efforts have integrated auxiliary modalities (e.g., optical flow) to introduce physics priors via joint training with video appearance. However, these methods have three main limitations: (1) they do not distinguish the different motion patterns of different entity types; (2) joint modeling of visual and auxiliary modalities can cause capacity conflicts and weaken the pretrained visual prior; and (3) auxiliary modalities may accumulate errors during inference. To address these issues, we propose VPT, a fine-tuning framework for improving physical consistency in video diffusion models. VPT introduces a role-aware signal that groups entities into agents, controlled objects, passive objects, and background, so that different physical roles can be modeled more clearly. We further propose a modality-decoupled denoising strategy, where the visual and auxiliary channels are assigned independent noise levels. Together with a loss-weight decay strategy, this design makes auxiliary modalities serve as soft constraints rather than strong dependencies, mitigating recursive prediction errors during inference. We also introduce cross-step auto-guidance to further strengthen physical dynamics. Experiments show that VPT improves physical consistency while preserving visual quality, achieving relative gains of 39.4% in SA and 17.9% in PC on VideoPhy benchmark over Wan2.1-T2V-1.3B, and consistent improvements on VideoPhy-2 benchmark. The project page is available at https://tom-zgt.github.io/VPT.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 26 of its 34 references that are in the map. First posted on arXiv 6 Jul 2026.
Builds on
26 of 34 references.
Closest ancestors · 12
Same direction, newest first
- 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models29 May
- 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding15 Apr
- 2025Goku: Flow Based Video Generative Foundation Models7 Feb
- 2024Autoregressive Video Generation without Vector Quantization18 Dec
- 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation30 Nov
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
- 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
- 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
- 2022Video Diffusion Models7 Apr
Other references · 14
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
- 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
- 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
- 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
- 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
- 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
- 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models
- 2026SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models
- 2026The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show
- 2026V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising
- 2025Temporal-Consistent Video Restoration with Pre-trained Diffusion Models
- 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusion
- 2024Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction
- 2026Physics in 2-Steps: Locking Motion Priors Before Visual Refinement Erases Them