mlsanityData to 14 Oct 2026

Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text

arXiv 2406.17601 ↗First posted 25 Jun 2024cs.CVcited by 50

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent advancements in 3D generation have leveraged synthetic datasets with ground truth 3D assets and predefined cameras. However, the potential of adopting real-world datasets, which can produce significantly more realistic 3D scenes, remains largely unexplored. In this work, we delve into the key challenge of the complex and scene-specific camera trajectories found in real-world captures. We introduce Director3D, a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the Cinematographer, to model the distribution of camera trajectories based on textual descriptions. (2) Next, a Gaussian-driven Multi-view Latent Diffusion Model serves as the Decorator, modeling the image sequence distribution given the camera trajectories and texts. This model, fine-tuned from a 2D diffusion model, directly generates pixel-aligned 3D Gaussians as an immediate 3D scene representation for consistent denoising. (3) Lastly, the 3D Gaussians are refined by a novel SDS++ loss as the Detailer, which incorporates the prior of the 2D diffusion model. Extensive experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 80 of its 93 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

80 of 93 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion16 May
  2. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion10 Apr
  3. 2023VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder18 Dec
  4. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation14 Dec
  5. 2023Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting8 Dec
  6. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  7. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  8. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models12 Oct
  9. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generation4 Oct
  10. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  11. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  12. 2023Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views27 Aug
11 more
  1. 2023HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation30 Jul
  2. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  3. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  4. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields19 May
  5. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  7. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models12 Dec
  8. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  9. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  10. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  11. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
38 more
  1. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  2. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  3. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  6. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  7. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  8. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  9. 20242D Gaussian Splatting for Geometrically Accurate Radiance Fieldscited by 1,076
  10. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  11. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  12. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  13. 2023Scaffold-GS: Structured 3D Gaussians for View-Adaptive Renderingcited by 695
  14. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 684
  15. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  16. 2023SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Renderingcited by 616
  17. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 566
  18. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
  19. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  20. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  21. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 476
  22. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  23. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 364
  24. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 349
  25. 2023Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesiscited by 344
  26. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  27. 2023Text-to-3D using Gaussian Splattingcited by 299
  28. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  29. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  30. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 231
  31. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 220
  32. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
  33. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 212
  34. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  35. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  36. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  37. 2023Noise-Free Score Distillationcited by 99
  38. 2024GSDF: 3DGS Meets SDF for Improved Rendering and Reconstructioncited by 83

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 31; 2026: 15.

50 papers.

Sort:

By area:

  1. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 94
  2. 2024VistaDream: Sampling multiview consistent images for single-view scene reconstructioncited by 38
  3. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  4. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  5. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generationcited by 34
  6. 2024MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Modelcited by 32
  7. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 32
  8. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 31
  9. 20253D Scene Generation: A Surveycited by 24
  10. 2025Drag Your Gaussian: Effective Drag-Based Editing with Score Distillation for 3D Gaussian Splattingcited by 22
  11. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
  12. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 19
38 more
  1. 2025GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photographycited by 18
  2. 2025SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steeringcited by 15
  3. 2025AR-1-to-3: Single Image to Consistent 3D Object Generation via Next-View Predictioncited by 15
  4. 2025VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modelingcited by 14
  5. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  6. 2025Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generatorcited by 11
  7. 2025DeOcc-1-to-3: 3D De-Occlusion from a Single Image via Self-Supervised Multi-View Diffusioncited by 10
  8. 20254DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Modelscited by 10
  9. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creationcited by 8
  10. 20253DOT: Texture Transfer for 3DGS Objects from a Single Reference Imagecited by 7
  11. 2026ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creationcited by 7
  12. 2025EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusioncited by 6
  13. 2025XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decodingcited by 6
  14. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 6
  15. 2026OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Datacited by 6
  16. 2025Generating 360deg Video is What You Need For a 3D Scenecited by 5
  17. 2025LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generationcited by 5
  18. 2025AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning in 4D Scenescited by 4
  19. 2025LAMP: Language-Assisted Motion Planning for Controllable Video Generationcited by 4
  20. 2025OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesiscited by 4
  21. 2025Generate Aligned Anomaly: Region-Guided Few-Shot Anomaly Image-Mask Pair Synthesis for Industrial Inspectioncited by 3
  22. 2025GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generationcited by 3
  23. 2026Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scalecited by 3
  24. 2026TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxycited by 3
  25. 2025LensCraft: Your Professional Virtual Cinematographercited by 2
  26. 2025RoomPlanner: Explicit Layout Planner for Easier LLM-Driven 3D Room Generationcited by 2
  27. 2026OneWorld: Taming Scene Generation with 3D Unified Representation Autoencodercited by 2
  28. 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generationcited by 2
  29. 2025Taking Language Embedded 3D Gaussian Splatting into the Wildcited by 1
  30. 2025FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillationcited by 1
  31. 2025CustomX: Unified Character, Action, and Scene Customization in Video World Modelscited by 1
  32. 2026Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generationcited by 1
  33. 2026JointEdit3D: Feed-Forward 3D Scene Editing in a Unified Latent Spacecited by 1
  34. 2025Interactive Video Generation via Domain Adaptationcited by 0
  35. 2026CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-Centric 3D Scene Generationcited by 0
  36. 2026CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labelscited by 0
  37. 2026ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptionscited by 0
  38. 2026Unveiling the Value of Motion for Cinematic Camera Trajectoriescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20243DEgo: 3D Editing on the Go!
  2. 20243D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
  3. 2024Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models
  4. 2024LT3SD: Latent Trees for 3D Scene Diffusion
  5. 2024Text-to-3D Generation by 2D Editing
  6. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenes
  7. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
  8. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models