mlsanityData to 14 Oct 2026

FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

arXiv 2406.16863 ↗First posted 24 Jun 2024cs.CVcited by 56

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomenons and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger video generation with controllable trajectories. Equipped with these designs, users have the flexibility to provide trajectories manually or opt for trajectories automatically generated by the LLM trajectory planner. Extensive experiments validate the efficacy of our approach in enhancing the trajectory controllability of video diffusion models.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 47 of its 53 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

47 of 53 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024FlexiFilm: Long Video Generation with Flexible Conditions29 Apr
  2. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  3. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion5 Feb
  4. 2024Lumiere: A Space-Time Diffusion Model for Video Generation23 Jan
  5. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation31 Dec
  6. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Models12 Dec
  7. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motion7 Dec
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  10. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  11. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  12. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
9 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  2. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  3. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis27 Apr
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  8. 2022Flexible Diffusion Modeling of Long Videos23 May
  9. 2022Video Diffusion Models7 Apr

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2013Auto-Encoding Variational Bayescited by 4,053
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  10. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  12. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
14 more
  1. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  2. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  3. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  4. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  5. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  6. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  7. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 128
  8. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  9. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 90
  10. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  11. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  12. 2023DragVideo: Interactive Drag-style Video Editingcited by 40
  13. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  14. 2024Move Anything with Layered Scene Diffusioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 29; 2026: 19.

56 papers.

Sort:

By area:

  1. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  3. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  4. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  5. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  6. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 61
  7. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  8. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  9. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  10. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  11. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 34
  12. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
38 more
  1. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  2. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 20
  3. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  4. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  5. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  7. 2025MotionV2V: Editing Motion in a Videocited by 14
  8. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 14
  9. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  10. 2025AnyI2V: Animating Any Conditional Image with Motion Controlcited by 11
  11. 2024CPA: Camera-pose-awareness Diffusion Transformer for Video Generationcited by 10
  12. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  13. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  14. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 10
  15. 2025TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixelscited by 7
  16. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  17. 2026Evolution of Video Generative Foundationscited by 7
  18. 2024Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversioncited by 6
  19. 2025C-Drag: Chain-of-Thought Driven Motion Controller for Video Generationcited by 6
  20. 2025Conditional Video Generation for High-Efficiency Video Compressioncited by 5
  21. 2025DiTraj: training-free trajectory control for video diffusion transformercited by 5
  22. 2025Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!cited by 5
  23. 2026FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidancecited by 5
  24. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  25. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 4
  26. 2025Enabling Versatile Controls for Video Diffusion Modelscited by 3
  27. 2025NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaborationcited by 3
  28. 2026EverAnimate: Minute-Scale Human Animation via Latent Flow Restorationcited by 3
  29. 2026Robot-Factored World Models via Robot Renderingcited by 3
  30. 2025When Distillation Breaks Motion Control: Restoring Generative Trajectories for Fast Video Generatorscited by 2
  31. 2025DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videoscited by 2
  32. 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generationcited by 1
  33. 2026Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Modelscited by 1
  34. 2026Training-free Motion Factorization for Compositional Video Generationcited by 1
  35. 2026NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignmentcited by 1
  36. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generationcited by 0
  37. 2025Interactive Video Generation via Domain Adaptationcited by 0
  38. 2026MotionAdapter: Video Motion Transfer via Content-Aware Attention Customizationcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation
  2. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models
  3. 2026PartiCam: Camera Controlled Video Generation with Reward Guidance
  4. 2025FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation
  5. 2026Efficient Video Diffusion Models: Advancements and Challenges
  6. 2023TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models
  7. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  8. 2024TVG: A Training-free Transition Video Generation Method with Diffusion Models