mlsanityData to 7 Oct 2026

VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models

arXiv 2505.23656 ↗First posted 29 May 2025cs.CVcited by 88

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advancements in text-to-video (T2V) diffusion models have enabled high-fidelity and realistic video synthesis. However, current T2V models often struggle to generate physically plausible content due to their limited inherent ability to accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag significantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called VideoREPA, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enable more physics-plausible generation. Specifically, we introduce the Token Relation Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft guidance suitable for finetuning powerful pre-trained T2V models, a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, CogVideoX, achieving significant improvement on relevant benchmarks and demonstrating a strong capacity for generating videos consistent with intuitive physics. More video results are available at https://videorepa.github.io/.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 50 of its 61 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

50 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformer15 Apr
  2. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation30 Nov
  3. 2024Phy124: Fast Physics-Driven 4D Content Generation from a Single Image11 Sep
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  5. 2024MotionCraft: Physics-based Zero-Shot Video Generation22 May
  6. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  8. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  9. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct

Other references · 41

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  7. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  10. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  11. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  12. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
29 more
  1. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  2. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  3. 2023Generating Images with Multimodal Language Modelscited by 259
  4. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  5. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 212
  6. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  7. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  8. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  9. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  10. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  11. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  12. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  13. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  14. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  15. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  16. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  17. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  18. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  19. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 82
  20. 2025Intuitive physics understanding emerges from self-supervised pretraining on natural videoscited by 66
  21. 2018ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stackingcited by 55
  22. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  23. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  24. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 29
  25. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 29
  26. 2025Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Thinkcited by 12
  27. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  28. 2025Towards Physical Understanding in Video Generation: A 3D Point Regularization Approachcited by 8
  29. 2025LEGO-Motion: Learning-Enhanced Grids with Occupancy Instance Modeling for Class-Agnostic Motion Predictioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 66.

88 papers.

Sort:

By area:

  1. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  2. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  3. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  4. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  5. 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Modelscited by 34
  6. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  7. 2026Transition Matching Distillation for Fast Video Generationcited by 27
  8. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 25
  9. 2026Improved Baselines with Representation Autoencoderscited by 24
  10. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  11. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  12. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Modelscited by 13
38 more
  1. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 12
  2. 2025ProPhy: Progressive Physical Alignment for Dynamic World Simulationcited by 11
  3. 2026Representation Alignment for Just Image Transformers is not Easier than You Thinkcited by 11
  4. 2026Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generationcited by 11
  5. 2026RefAlign: Representation Alignment for Reference-to-Video Generationcited by 10
  6. 2026Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamicscited by 10
  7. 2026NEWTON: Agentic Planning for Physically Grounded Video Generationcited by 10
  8. 2026SRA 2: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Trainingcited by 8
  9. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 8
  10. 2026SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editingcited by 8
  11. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 8
  12. 2025LayerSync: Self-aligning Intermediate Layerscited by 7
  13. 2025Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspectivecited by 7
  14. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  15. 2026GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulationcited by 7
  16. 2026PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulationcited by 7
  17. 2025DeRA: Decoupled Representation Alignment for Video Tokenizationcited by 6
  18. 2026DreamWorld: Unified World Modeling in Video Generationcited by 6
  19. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  20. 2026Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generationcited by 6
  21. 2026FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editingcited by 5
  22. 2026PhysVid: Physics Aware Local Conditioning for Generative Video Modelscited by 5
  23. 2025MATRIX: Mask Track Alignment for Interaction-aware Video Generationcited by 4
  24. 2025PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Modelscited by 4
  25. 2026LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generationcited by 4
  26. 2026VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimizationcited by 4
  27. 2026MaineCoon: Pursuing A Real-Time Audio-Visual Social World Modelcited by 4
  28. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  29. 2025View-Consistent Diffusion Representations for 3D-Consistent Video Generationcited by 3
  30. 2025Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinementcited by 3
  31. 2025Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for nowcited by 3
  32. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  33. 2026Motion Forcing: A Decoupled Framework for Robust Video Generation in Motion Dynamicscited by 3
  34. 2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 3
  35. 2026VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoderscited by 3
  36. 2026V-RAE: Rethinking Video Latent Spaces for Generationcited by 3
  37. 2026V-Co: A Closer Look at Visual Representation Alignment via Co-Denoisingcited by 2
  38. 2026MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modelingcited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026How Do Video Foundation Models Encode Intuitive Physics? Probing Across Pretraining Paradigms
  2. 2025Improving the Physics of Video Generation with VJEPA-2 Reward Signal
  3. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusion
  4. 2026Causal Physics Steering in Video World Models via Concept Activation Vectors
  5. 2026VIPER: Visual In-Context Physics Reasoning for Physically Plausible Video Generation
  6. 2026Interpreting Physics in Video World Models
  7. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suite
  8. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation