mlsanityData to 7 Oct 2026

Tora: Trajectory-oriented Diffusion Transformer for Video Generation

arXiv 2407.21705 ↗First posted 31 Jul 2024cs.CVcited by 161

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that concurrently integrates textual, visual, and trajectory conditions, thereby enabling scalable video generation with effective motion guidance. Specifically, Tora consists of a Trajectory Extractor (TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser (MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D motion compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos that accurately follow designated trajectories. Our design aligns seamlessly with DiT's scalability, allowing precise control of video content's dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate that Tora excels in achieving high motion fidelity compared to the foundational DiT model, while also accurately simulating the complex movements of the physical world. Code is made available at https://github.com/alibaba/Tora .

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 44 of its 59 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

44 of 59 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024ReVideo: Remake a Video with Motion and Content Control22 May
  2. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models7 May
  3. 2024Lumiere: A Space-Time Diffusion Model for Video Generation23 Jan
  4. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation31 Dec
  5. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videos25 Dec
  6. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models1 Dec
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  8. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  10. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  11. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
2 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Video Diffusion Models7 Apr

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
18 more
  1. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  2. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  3. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  4. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  5. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  6. 2020Virtual KITTI 2cited by 345
  7. 2022MAGVIT: Masked Generative Video Transformercited by 341
  8. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  9. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  10. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  11. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  12. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  13. 2023Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereocited by 170
  14. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  15. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  16. 2022ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wildcited by 90
  17. 2019Learning Multi-Human Optical Flowcited by 26
  18. 2023PUCA: Patch-Unshuffle and Channel Attention for Enhanced Self-Supervised Image Denoisingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 78; 2026: 60.

161 papers.

Sort:

By area:

  1. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  2. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  3. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  4. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  5. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  6. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  7. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  8. 2025Controllable Video Generation: A Surveycited by 93
  9. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  10. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  11. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 73
  12. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
38 more
  1. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  2. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  3. 2025SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restorationcited by 62
  4. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  5. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  6. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  7. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  8. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  9. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generationcited by 41
  10. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  11. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  12. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  13. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  14. 2025HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generationcited by 38
  15. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  16. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 33
  17. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  18. 2025Dynamic Camera Poses and Where to Find Themcited by 26
  19. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  20. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  21. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 22
  22. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  23. 2024Boosting Camera Motion Control for Video Diffusion Transformerscited by 20
  24. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  25. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  26. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 19
  27. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 18
  28. 2024MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generationcited by 16
  29. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  30. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  31. 2025SketchVideo: Sketch-based Video Generation and Editingcited by 15
  32. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  33. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  34. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  35. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
  36. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  37. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Modelscited by 13
  38. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
  2. 2025GalaxyDiT: Efficient Video Generation with Guidance Alignment and Adaptive Proxy in Diffusion Transformers
  3. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling
  4. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  5. 2023TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models
  6. 2025DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generation
  7. 2025High-Fidelity and Long-Duration Human Image Animation with Diffusion Transformer
  8. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer