mlsanityData to 7 Oct 2026

TrackGo: A Flexible and Efficient Method for Controllable Video Generation

arXiv 2408.11475 ↗First posted 21 Aug 2024cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation. This method offers users with a flexible and precise mechanism for manipulating video content. We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers of a pretrained video generation model. This design leverages our observation that the attention map of these layers can accurately activate regions corresponding to motion in videos. Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and ObjMC scores.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

33 of 40 references.

References · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  10. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  11. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
21 more
  1. 2023ModelScope Text-to-Video Technical Reportcited by 651
  2. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  3. 2023CoTracker: It is Better to Track Togethercited by 516
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  5. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  6. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  7. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  8. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  9. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  10. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  11. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  12. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  13. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  14. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  15. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  16. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  17. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  18. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  19. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  20. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 56
  21. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 16.

39 papers.

Sort:

By area:

  1. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  3. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  4. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  5. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  6. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  7. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  8. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  9. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  10. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  11. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  12. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
27 more
  1. 2024I2VControl: Disentangled and Unified Video Motion Synthesis Controlcited by 19
  2. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  3. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  4. 2024Motion Modes: What Could Happen Next?cited by 11
  5. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  6. 2025AnyI2V: Animating Any Conditional Image with Motion Controlcited by 11
  7. 2024CPA: Camera-pose-awareness Diffusion Transformer for Video Generationcited by 10
  8. 2025Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initializationcited by 9
  9. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 9
  10. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  11. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  12. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  13. 2026CtrlAttack: A Unified Attack on World-Model Control in Diffusion Modelscited by 6
  14. 2025Model as a Game: On Numerical and Spatial Consistency for Generative Gamescited by 5
  15. 2025Conditional Video Generation for High-Efficiency Video Compressioncited by 5
  16. 2026FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidancecited by 5
  17. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  18. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 3
  19. 2026Robot-Factored World Models via Robot Renderingcited by 3
  20. 2025Learning to Generate Rigid Body Interactions with Video Diffusion Modelscited by 2
  21. 2025VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videoscited by 0
  22. 2026Search2Motion: Training-Free Object-Level Motion Control via Attention-Consensus Searchcited by 0
  23. 2026ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generationcited by 0
  24. 2026Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representationscited by 0
  25. 2026Physically Plausible Video Generation via Visual-Semantic Chain-of-Events Conditioningcited by 0
  26. 2026What Moves? Localized Motion Representations for Compositional Scene Controlcited by 0
  27. 20264Director: Controlling Video World Models with Rigid 3D Geometrycited by 0