# The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show

arXiv: https://arxiv.org/abs/2606.05328 · HTML: https://mlsanity.com/paper/2606.05328/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 14 of 14)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [Intuitive physics understanding emerges from self-supervised pretraining on natural videos](https://mlsanity.com/paper/2502.11831/) (2025)
- [SemanticMoments: Training-Free Motion Similarity via Third Moment Features](https://mlsanity.com/paper/2602.09146/) (2026)

## Built on by (most cited first, 8 of 8)

- [PhysWeep: Does a Video Generator Realize the Physics You Ask For?](https://mlsanity.com/paper/2609.06207/) (2026)
- [Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning](https://mlsanity.com/paper/2607.17673/) (2026)
- [A Chosen Future Can Still Be Rewritten: Causal Writability in Video Models](https://mlsanity.com/paper/2609.15980/) (2026)
- [Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video](https://mlsanity.com/paper/2609.38819/) (2026)
- [Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models](https://mlsanity.com/paper/2610.03154/) (2026)
- [Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object](https://mlsanity.com/paper/2610.07355/) (2026)
- [When Generative Models Act as Simulators](https://mlsanity.com/paper/2610.12847/) (2026)
- [StateCraft: Augmenting Video World Models with Abstract-State Reasoning](https://mlsanity.com/paper/2610.19795/) (2026)

## Abstract (arXiv)

Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model's intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.