# Emergent Temporal Correspondences from Video Diffusion Transformers

arXiv: https://arxiv.org/abs/2506.17220 · HTML: https://mlsanity.com/paper/2506.17220/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 81 of its 88 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 81)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)

## Built on by (most cited first, 30 of 31)

- [Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention](https://mlsanity.com/paper/2602.01801/) (2026)
- [CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Models](https://mlsanity.com/paper/2512.03045/) (2025)
- [Point Prompting: Counterfactual Tracking with Video Diffusion Models](https://mlsanity.com/paper/2510.11715/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking](https://mlsanity.com/paper/2605.12587/) (2026)
- [The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction](https://mlsanity.com/paper/2606.30308/) (2026)
- [Taming generative video models for zero-shot optical flow extraction](https://mlsanity.com/paper/2507.09082/) (2025)
- [MATRIX: Mask Track Alignment for Interaction-aware Video Generation](https://mlsanity.com/paper/2510.07310/) (2025)
- [Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence](https://mlsanity.com/paper/2512.04619/) (2025)
- [V-Warper: Appearance-Consistent Video Diffusion Personalization via Value Warping](https://mlsanity.com/paper/2512.12375/) (2025)
- [Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking](https://mlsanity.com/paper/2512.20606/) (2025)
- [MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation](https://mlsanity.com/paper/2606.26087/) (2026)
- [Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms](https://mlsanity.com/paper/2609.23658/) (2026)
- [BachVid: Training-Free Video Generation with Consistent Background and Character](https://mlsanity.com/paper/2510.21696/) (2025)
- [Moaw: Unleashing Motion Awareness for Video Diffusion Models](https://mlsanity.com/paper/2601.12761/) (2026)
- [CORAL: Correspondence Alignment for Improved Virtual Try-On](https://mlsanity.com/paper/2602.17636/) (2026)
- [Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers](https://mlsanity.com/paper/2603.02919/) (2026)
- [TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation](https://mlsanity.com/paper/2603.23487/) (2026)
- [Fast 4D Mesh Generation by Spatio-Temporal Attention Chains](https://mlsanity.com/paper/2605.19786/) (2026)
- [Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning](https://mlsanity.com/paper/2605.30231/) (2026)
- [Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos](https://mlsanity.com/paper/2511.19936/) (2025)
- [M2P: Improving Visual Foundation Models with Mask-to-Point Weakly-Supervised Learning for Dense Point Tracking](https://mlsanity.com/paper/2603.17813/) (2026)
- [DA-Flow: Degradation-Aware Optical Flow Estimation with Diffusion Models](https://mlsanity.com/paper/2603.23499/) (2026)
- [Geometry-Aware Representation Denoising for Robust Multi-view 3D Reconstruction](https://mlsanity.com/paper/2605.26230/) (2026)
- [QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformers](https://mlsanity.com/paper/2607.01869/) (2026)
- [Controlling Motion Transfer in Diffusion Transformers via Attention Heads](https://mlsanity.com/paper/2607.11081/) (2026)
- [DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents](https://mlsanity.com/paper/2608.00486/) (2026)
- [ASTRA: Asynchronous Spatio-Temporal Reconstruction via Trajectory Alignment](https://mlsanity.com/paper/2608.02006/) (2026)
- [Temporal-Attention Head Specialization During Video Diffusion Training](https://mlsanity.com/paper/2609.31654/) (2026)
- [Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video](https://mlsanity.com/paper/2609.38819/) (2026)

## Abstract (arXiv)

Recent advancements in video diffusion models based on Diffusion Transformers (DiTs) have achieved remarkable success in generating temporally coherent videos. Yet, a fundamental question persists: how do these models internally establish and represent temporal correspondences across frames? We introduce DiffTrack, the first quantitative analysis framework designed to answer this question. DiffTrack constructs a dataset of prompt-generated video with pseudo ground-truth tracking annotations and proposes novel evaluation metrics to systematically analyze how each component within the full 3D attention mechanism of DiTs (e.g., representations, layers, and timesteps) contributes to establishing temporal correspondences. Our analysis reveals that query-key similarities in specific, but not all, layers play a critical role in temporal matching, and that this matching becomes increasingly prominent during the denoising process. We demonstrate practical applications of DiffTrack in zero-shot point tracking, where it achieves state-of-the-art performance compared to existing vision foundation and self-supervised video models. Further, we extend our findings to motion-enhanced video generation with a novel guidance method that improves temporal consistency of generated videos without additional training. We believe our work offers crucial insights into the inner workings of video DiTs and establishes a foundation for further research and applications leveraging their temporal understanding.