mlsanityData to 7 Oct 2026

Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer

arXiv 2405.14832 ↗First posted 23 May 2024cs.CVcited by 210

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Generating high-quality 3D assets from text and images has long been challenging, primarily due to the absence of scalable 3D representations capable of capturing intricate geometry distributions. In this work, we introduce Direct3D, a native 3D generative model scalable to in-the-wild input images, without requiring a multiview diffusion model or SDS optimization. Our approach comprises two primary components: a Direct 3D Variational Auto-Encoder (D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). D3D-VAE efficiently encodes high-resolution 3D shapes into a compact and continuous latent triplane space. Notably, our method directly supervises the decoded geometry using a semi-continuous surface sampling strategy, diverging from previous methods relying on rendered images as supervision signals. D3D-DiT models the distribution of encoded 3D latents and is specifically designed to fuse positional information from the three feature maps of the triplane latent, enabling a native 3D generative model scalable to large-scale 3D datasets. Additionally, we introduce an innovative image-to-3D generation pipeline incorporating semantic and pixel-level image conditions, allowing the model to produce 3D shapes consistent with the provided conditional image input. Extensive experiments demonstrate the superiority of our large-scale pre-trained Direct3D over previous image-to-3D approaches, achieving significantly better generation quality and generalization ability, thus establishing a new state-of-the-art for 3D content creation. Project page: https://nju-3dv.github.io/projects/Direct3D/.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 58 of its 68 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

58 of 68 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  2. 2024Compress3D: a Compressed Latent Space for 3D Generation from a Single Image20 Mar
  3. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors4 Mar
  4. 2023Mosaic-SDF for 3D Generative Models14 Dec
  5. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation2 Dec
  6. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D28 Nov
  7. 2023Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion27 Nov
  8. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion14 Nov
  9. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  10. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  11. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  12. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
10 more
  1. 2023DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation4 Jul
  2. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  4. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  5. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  7. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  8. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  10. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
24 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  3. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  6. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  7. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  8. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  9. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  10. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  11. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  12. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  13. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  14. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  15. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  16. 2022SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Viewscited by 175
  17. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  18. 2023PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Predictioncited by 165
  19. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  20. 2023Text-Guided Texturing by Synchronized Multi-View Diffusioncited by 86
  21. 2023Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Viewscited by 32
  22. 2024Make-A-Shape: a Ten-Million-scale 3D Shape Modelcited by 25
  23. 2024MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusioncited by 22
  24. 2024GetMesh: A Controllable Model for High-quality Mesh Generation and Manipulationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 94; 2026: 85.

210 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  2. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  3. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  4. 2025TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Modelscited by 204
  5. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  6. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  7. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 113
  8. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  9. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  10. 2025Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridgingcited by 95
  11. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoderscited by 93
  12. 2024LLaMA-Mesh: Unifying 3D Mesh Generation with Language Modelscited by 87
38 more
  1. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  2. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  3. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  4. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 76
  5. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  6. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 62
  7. 2025OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesioncited by 60
  8. 2025HoloPart: Generative 3D Part Amodal Segmentationcited by 57
  9. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 52
  10. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  11. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  12. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  13. 2025Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attentioncited by 43
  14. 2025VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Spacecited by 40
  15. 2025Matrix3D: Large Photogrammetry Model All-in-Onecited by 39
  16. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 38
  17. 2024DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenescited by 37
  18. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  19. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generationcited by 33
  20. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  21. 2025ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generationcited by 30
  22. 2025Unleashing Vecset Diffusion Model for Fast Shape Generationcited by 25
  23. 2025DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusioncited by 23
  24. 2025FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusioncited by 22
  25. 2025Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuningcited by 21
  26. 2025UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generationcited by 20
  27. 2025Nautilus: Locality-aware Autoencoder for Scalable Mesh Generationcited by 18
  28. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
  29. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Predictioncited by 16
  30. 2026MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generationcited by 16
  31. 2024G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformercited by 15
  32. 2025AvatarArtist: Open-Domain 4D Avatarizationcited by 15
  33. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  34. 2025SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Modelcited by 15
  35. 2024Turbo3D: Ultra-fast Text-to-3D Generationcited by 14
  36. 2025Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generationcited by 14
  37. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  38. 2025CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generationcited by 14

50 most cited of 210 All 210 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffTF++: 3D-aware Diffusion Transformer for Large-Vocabulary 3D Generation
  2. 2023Large-Vocabulary 3D Diffusion Model with Transformer
  3. 2024DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data
  4. 2025LATTICE: Democratize High-Fidelity 3D Generation at Scale
  5. 2024LT3SD: Latent Trees for 3D Scene Diffusion
  6. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
  7. 2026Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
  8. 2024Taming Feed-forward Reconstruction Models as Latent Encoders for 3D Generative Models