mlsanityData to 7 Oct 2026

Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation

arXiv 2412.06016 ↗First posted 8 Dec 2024cs.CV cs.AI cs.LGcited by 42

On the map: Diffusion models → Video generation

Abstract (arXiv)

While recent foundational video generators produce visually rich output, they still struggle with appearance drift, where objects gradually degrade or change inconsistently across frames, breaking visual coherence. We hypothesize that this is because there is no explicit supervision in terms of spatial tracking at the feature level. We propose Track4Gen, a spatially aware video generator that combines video diffusion loss with point tracking across frames, providing enhanced spatial supervision on the diffusion features. Track4Gen merges the video generation and point tracking tasks into a single network by making minimal changes to existing video generation architectures. Using Stable Video Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify video generation and point tracking, which are typically handled as separate tasks. Our extensive evaluations show that Track4Gen effectively reduces appearance drift, resulting in temporally stable and visually coherent video generation. Project page: hyeonho99.github.io/track4gen

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 68 of its 81 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

68 of 81 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024HARIVO: Harnessing Text-to-Image Models for Video Generation10 Oct
  2. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence4 Dec
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  6. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  9. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  10. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  12. 2022Video Diffusion Models7 Apr

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
38 more
  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2017Decoupled Weight Decay Regularizationcited by 1,722
  4. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  5. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  7. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  8. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  9. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  10. 2023CoTracker: It is Better to Track Togethercited by 516
  11. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  12. 2023Emergent Correspondence from Image Diffusioncited by 437
  13. 2022Kubric: A scalable dataset generatorcited by 373
  14. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  15. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  16. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  17. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  18. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  19. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  20. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  21. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  22. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  23. 2023Tracking Everything Everywhere All at Oncecited by 190
  24. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 178
  25. 2023AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolationcited by 177
  26. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  27. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  28. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  29. 2024FeatUp: A Model-Agnostic Framework for Features at Any Resolutioncited by 114
  30. 2023Diffusion Model as Representation Learnercited by 95
  31. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
  32. 2018Creatures great and SMAL: Recovering the shape and motion of animals from videocited by 84
  33. 2024Local All-Pair Correspondence for Point Trackingcited by 77
  34. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  35. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  36. 2023Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Featurescited by 55
  37. 2024Towards A Better Metric for Text-to-Video Generationcited by 50
  38. 2018The 2018 DAVIS Challenge on Video Object Segmentationcited by 47

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 15.

42 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  3. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  4. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  5. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 30
  6. 2025Aligning Text to Image in Diffusion Models is Easier Than You Thinkcited by 21
  7. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  8. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 14
  9. 2025VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learningcited by 12
  10. 2025SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Timecited by 9
  11. 2026VOID: Video Object and Interaction Deletioncited by 9
  12. 2025JOG3R: Towards 3D-Consistent Video Generatorscited by 8
30 more
  1. 2025Towards Physical Understanding in Video Generation: A 3D Point Regularization Approachcited by 8
  2. 2025CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Modelscited by 8
  3. 2025Target-Aware Video Diffusion Modelscited by 7
  4. 2026ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generationcited by 6
  5. 2025MoSA: Motion-Coherent Human Video Generation via Structure-Appearance Decouplingcited by 4
  6. 2025MATRIX: Mask Track Alignment for Interaction-aware Video Generationcited by 4
  7. 2025MV-TAP: Tracking Any Point in Multi-View Videoscited by 4
  8. 2025AnthroTAP: Learning Point Tracking with Real-World Motioncited by 3
  9. 2025View-Consistent Diffusion Representations for 3D-Consistent Video Generationcited by 3
  10. 2025WorldReel: 4D Video Generation with Consistent Geometry and Motion Modelingcited by 3
  11. 2025Improving Video Diffusion Transformer Training by Multi-Feature Fusion and Alignment from Self-Supervised Vision Encoderscited by 2
  12. 2026SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuningcited by 2
  13. 2026Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Modelscited by 2
  14. 2026LooseControlVideo: Directorial Video Control using Spatial Blockingcited by 2
  15. 2025Tracking-Guided 4D Generation: Foundation-Tracker Motion Priors for 3D Model Animationcited by 1
  16. 2025Structure From Tracking: Distilling Structure-Preserving Motion for Video Generationcited by 1
  17. 2026CORAL: Correspondence Alignment for Improved Virtual Try-Oncited by 1
  18. 2026TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolationcited by 1
  19. 2026Improving Human Image Animation via Semantic Representation Alignmentcited by 1
  20. 2026Fast 4D Mesh Generation by Spatio-Temporal Attention Chainscited by 1
  21. 2026Tempered Self-Similarity Alignment for Physically Plausible Video Generationcited by 1
  22. 2026Gen4U: Unifying Video Generation and Understanding via Diffusioncited by 1
  23. 2025SAM2Auto: Auto Annotation Using FLASHcited by 0
  24. 2025Align Your Tangent: Training Better Consistency Models via Manifold-Aligned Tangentscited by 0
  25. 2025Flow and Depth Assisted Video Prediction with Latent Transformercited by 0
  26. 2025Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervisioncited by 0
  27. 2026Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRAcited by 0
  28. 20263DPhysVideo: Consistency-Guided Flow SDE for Video Generation via 3D Scene Reconstruction and Physical Simulationcited by 0
  29. 2026DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latentscited by 0
  30. 2026MotionSpec: Spectral Trajectory Supervision for Motion-Consistent Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models
  2. 2025Point Prompting: Counterfactual Tracking with Video Diffusion Models
  3. 2026Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks
  4. 2026Moaw: Unleashing Motion Awareness for Video Diffusion Models
  5. 2026MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation
  6. 2026Go-with-the-Track: Video Compositing and Motion Control with Point Tracking
  7. 2023MoVideo: Motion-Aware Video Generation with Diffusion Models
  8. 2026PE-Field 4D: Video Generation Models as Canvas