mlsanityData to 7 Oct 2026

TrailBlazer: Trajectory Control for Diffusion-Based Video Generation

arXiv 2401.00896 ↗First posted 31 Dec 2023cs.CVcited by 90

On the map: Diffusion models → Video generation

Abstract (arXiv)

Within recent approaches to text-to-video (T2V) generation, achieving controllability in the synthesized video is often a challenge. Typically, this issue is addressed by providing low-level per-frame guidance in the form of edge maps, depth maps, or an existing video to be altered. However, the process of obtaining such guidance can be labor-intensive. This paper focuses on enhancing controllability in video synthesis by employing straightforward bounding boxes to guide the subject in various ways, all without the need for neural network training, finetuning, optimization at inference time, or the use of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a pre-trained (T2V) model, and easy to implement. The subject is directed by a bounding box through the proposed spatial and temporal attention map editing. Moreover, we introduce the concept of keyframing, allowing the subject trajectory and overall appearance to be guided by both a moving bounding box and corresponding prompts, without the need to provide a detailed mask. The method is efficient, with negligible additional computation relative to the underlying pre-trained model. Despite the simplicity of the bounding box guidance, the resulting motion is surprisingly natural, with emergent effects including perspective and movement toward the virtual camera as the box size increases.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 41 of its 63 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

41 of 63 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023MagicProp: Diffusion-based Video Editing via Motion-aware Appearance Propagation2 Sep
  2. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  7. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  8. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  11. 2022Flexible Diffusion Modeling of Long Videos23 May
  12. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
1 more
  1. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
16 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  6. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  7. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  8. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  9. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  10. 2022Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understandingcited by 102
  11. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  12. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidancecited by 69
  13. 2022MagicMix: Semantic Mixing with Diffusion Modelscited by 63
  14. 2020Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesiscited by 49
  15. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  16. 2021Mitigating the Bias of Centered Objects in Common Datasetscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 35; 2026: 28.

90 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  2. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  3. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  4. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  5. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  6. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  7. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  8. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  9. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  10. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  11. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  12. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
38 more
  1. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  2. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  3. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  4. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
  5. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 42
  6. 2024TrackGo: A Flexible and Efficient Method for Controllable Video Generationcited by 39
  7. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  8. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  9. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  10. 2024Motion Inversion for Video Customizationcited by 30
  11. 2025MotionPro: A Precise Motion Controller for Image-to-Video Generationcited by 29
  12. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  13. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generationcited by 25
  14. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  15. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  16. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  17. 2024MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Modelscited by 16
  18. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  19. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  20. 2024Intention-driven Ego-to-Exo Video Generationcited by 13
  21. 2024MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidancecited by 13
  22. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  23. 2025MotionV2V: Editing Motion in a Videocited by 13
  24. 2024EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolationcited by 12
  25. 2025TransPixeler: Advancing Text-to-Video Generation with Transparencycited by 12
  26. 2025Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generationcited by 12
  27. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 11
  28. 2025AnyI2V: Animating Any Conditional Image with Motion Controlcited by 11
  29. 2024Animate Your Motion: Turning Still Images into Dynamic Videoscited by 10
  30. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  31. 2024DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationshipscited by 9
  32. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  33. 2025LayerFlow: A Unified Model for Layer-aware Video Generationcited by 9
  34. 2025WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generationcited by 9
  35. 2026AdaGaR: Adaptive Gabor Representation for Dynamic Scene Reconstructioncited by 8
  36. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 7
  37. 2025The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Textcited by 7
  38. 2024Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversioncited by 6

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning
  2. 2025Resource-Efficient Motion Control for Video Generation via Dynamic Mask Guidance
  3. 2025Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization
  4. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  5. 2025Controllable Video Generation: A Survey
  6. 2025EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation
  7. 2023ControlVideo: Training-free Controllable Text-to-Video Generation
  8. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model