mlsanityData to 14 Oct 2026

3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation

arXiv 2412.07759 ↗First posted 10 Dec 2024cs.CVcited by 71

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper aims to manipulate multi-entity 3D motions in video generation. Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our approach is a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video quality degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable training data, we construct a 360-Motion Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions. Project page: http://fuxiao0719.github.io/projects/3dtrajmaster

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 54 of its 95 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

54 of 95 references.

References · 54

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
38 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  4. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  6. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  7. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  8. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  9. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  10. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  11. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  12. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  13. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  14. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  15. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
  16. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  17. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  18. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  19. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 201
  20. 2023MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyondcited by 185
  21. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  22. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  23. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  24. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 164
  25. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 157
  26. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  27. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  28. 2024World-Grounded Human Motion Recovery via Gravity-View Coordinatescited by 137
  29. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  30. 2024Training-free Camera Control for Video Generationcited by 122
  31. 2022Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Imagescited by 121
  32. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  33. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  34. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Modelscited by 86
  35. 2023CIRCLE: Capture In Rich Contextual Environmentscited by 79
  36. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
  37. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
  38. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 56

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 36; 2026: 33.

71 papers.

Sort:

By area:

  1. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  2. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  3. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 99
  4. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 98
  5. 2025Controllable Video Generation: A Surveycited by 98
  6. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 74
  7. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 62
  8. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 61
  9. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  10. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 41
  11. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  12. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 34
38 more
  1. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  2. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  3. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 26
  4. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 20
  5. 2025Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Controlcited by 18
  6. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  7. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  8. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  9. 2025MotionV2V: Editing Motion in a Videocited by 14
  10. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  11. 2025SCAIL: Towards Studio-Grade Character Animation via In-Context Learning of 3D-Consistent Pose Representationscited by 13
  12. 2025RealisMotion: Decomposed Human Motion Control and Video Generation in the World Spacecited by 12
  13. 2025Light-X: Generative 4D Video Rendering with Camera and Illumination Controlcited by 12
  14. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  15. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  16. 2025Taming Camera-Controlled Video Generation with Verifiable Geometry Rewardcited by 10
  17. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 10
  18. 2025OmniCam: Unified Multimodal Video Generation via Camera Controlcited by 9
  19. 2025SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulationcited by 9
  20. 2026Plenoptic Video Generationcited by 9
  21. 2026MemLearner: Learning to Query Context memory for Video World Modelscited by 9
  22. 2025FlexTraj: Image-to-Video Generation with Flexible Point Trajectory Controlcited by 6
  23. 2026FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidancecited by 5
  24. 2025Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generationcited by 4
  25. 2025AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning in 4D Scenescited by 4
  26. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  27. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 4
  28. 2026Pantheon360: Taming Digital Twin Generation via 3D-Aware 360deg Video Diffusioncited by 4
  29. 2025Enabling Versatile Controls for Video Diffusion Modelscited by 3
  30. 2025MTVCraft: Tokenizing 4D Motion for Arbitrary Character Animationcited by 3
  31. 2025Video4Spatial: Towards Visuospatial Intelligence with Context-Guided Video Generationcited by 3
  32. 2026Stereo World Model: Camera-Guided Stereo Video Generationcited by 3
  33. 2026AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customizationcited by 3
  34. 2026Holo-World: Unified Camera, Object and Weather Control for Video World Modelcited by 3
  35. 2025ChronosObserver: Taming 4D World with Hyperspace Diffusion Samplingcited by 2
  36. 2025DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videoscited by 2
  37. 2026Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshootingcited by 2
  38. 2026Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Trackscited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PoseTraj: Pose-Aware Trajectory Control in Video Diffusion
  2. 2024DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships
  3. 20264DStreamCtrl: Interactive Video Generation with Online 4D Control
  4. 2026World2Motion: Turning Video World Models into 3D Human Motion Generators
  5. 2025I2V3D: Controllable image-to-video generation with 3D guidance
  6. 2023MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion
  7. 2025DiTraj: training-free trajectory control for video diffusion transformer
  8. 2025Strong and Controllable 3D Motion Generation