mlsanityData to 7 Oct 2026

LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis

arXiv 2412.15214 ↗First posted 19 Dec 2024cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

The intuitive nature of drag-based interaction has led to its growing adoption for controlling object trajectories in image-to-video synthesis. Still, existing methods that perform dragging in the 2D space usually face ambiguity when handling out-of-plane movements. In this work, we augment the interaction with a new dimension, i.e., the depth dimension, such that users are allowed to assign a relative depth for each point on the trajectory. That way, our new interaction paradigm not only inherits the convenience from 2D dragging, but facilitates trajectory control in the 3D space, broadening the scope of creativity. We propose a pioneering method for 3D trajectory control in image-to-video synthesis by abstracting object masks into a few cluster points. These points, accompanied by the depth information and the instance information, are finally fed into a video diffusion model as the control signal. Extensive experiments validate the effectiveness of our approach, dubbed LeviTor, in precisely manipulating the object movements when producing photo-realistic videos from static images. Our code is available at: https://github.com/ant-research/LeviTor.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 62 of its 67 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

62 of 67 references.

References · 62

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2023Segment Anythingcited by 8,158
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
38 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  2. 2024Depth Anything V2cited by 1,385
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  7. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  8. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  9. 2023ModelScope Text-to-Video Technical Reportcited by 651
  10. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  11. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  12. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  13. 2023CoTracker: It is Better to Track Togethercited by 516
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  15. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  16. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  17. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  18. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  19. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  20. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  21. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  22. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  23. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  24. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  25. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  26. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  27. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  28. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  29. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  30. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  31. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 95
  32. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  33. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  34. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  35. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  36. 2023LooseControl: Lifting ControlNet for Generalized Depth Conditioningcited by 68
  37. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  38. 2024Image Sculpting: Precise Object Editing with 3D Geometry Controlcited by 45

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 16.

39 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  3. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  4. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 42
  5. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  6. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  7. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  8. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  9. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 22
  10. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  11. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 19
  12. 2025Motion-Aware Generative Frame Interpolationcited by 13
27 more
  1. 2025Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generationcited by 12
  2. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  3. 2025O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editingcited by 10
  4. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 9
  5. 2025Zo3T: Zero-Shot 3D-Aware Trajectory-Guided Image-to-Video Generation via Test-Time Trainingcited by 7
  6. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 7
  7. 2025The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Textcited by 7
  8. 2025FlexTraj: Image-to-Video Generation with Flexible Point Trajectory Controlcited by 6
  9. 2025Planning with Sketch-Guided Verification for Physics-Aware Video Generationcited by 6
  10. 2025DiTraj: training-free trajectory control for video diffusion transformercited by 5
  11. 2026FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidancecited by 5
  12. 2025PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherencecited by 4
  13. 2026WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memorycited by 4
  14. 2025Consistent and Controllable Image Animation with Motion Linear Diffusion Transformerscited by 3
  15. 2026TRACE: Object Motion Editing in Videos with First-Frame Trajectory Guidancecited by 3
  16. 2025DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videoscited by 2
  17. 2026Image-to-Video Diffusion: From Foundations to Open Frontierscited by 1
  18. 2026Go-with-the-Track: Video Compositing and Motion Control with Point Trackingcited by 1
  19. 2026Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generationcited by 1
  20. 2025YingVideo-MV: Music-Driven Multi-Stage Video Generationcited by 0
  21. 2026Tora3: Trajectory-Guided Audio-Video Generation with Physical Coherencecited by 0
  22. 2026Controllable Dynamic 3D Shape Generation via 3D Trajectories and Textcited by 0
  23. 2026Geometry-Instructed Video Editingcited by 0
  24. 2026TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Controlcited by 0
  25. 2026FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Controlcited by 0
  26. 2026LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillationcited by 0
  27. 20264Director: Controlling Video World Models with Rigid 3D Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships
  2. 2025PoseTraj: Pose-Aware Trajectory Control in Video Diffusion
  3. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  4. 2025C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation
  5. 2023DragVideo: Interactive Drag-style Video Editing
  6. 20264DStreamCtrl: Interactive Video Generation with Online 4D Control
  7. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation
  8. 2026Reproducing DragDiffusion: Interactive Point-Based Editing with Diffusion Models