mlsanityData to 7 Oct 2026

VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder

arXiv 2312.11459 ↗First posted 18 Dec 2023cs.CVcited by 39

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This paper introduces a pioneering 3D volumetric encoder designed for text-to-3D generation. To scale up the training data for the diffusion model, a lightweight network is developed to efficiently acquire feature volumes from multi-view images. The 3D volumes are then trained on a diffusion model for text-to-3D generation using a 3D U-Net. This research further addresses the challenges of inaccurate object captions and high-dimensional feature volumes. The proposed model, trained on the public Objaverse dataset, demonstrates promising outcomes in producing diverse and recognizable samples from text prompts. Notably, it empowers finer control over object part characteristics through textual cues, fostering model creativity by seamlessly combining multiple concepts within a single object. This research significantly contributes to the progress of 3D generation by introducing an efficient, flexible, and scalable representation methodology.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 47 of its 52 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

47 of 52 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023HiFi-123: Towards High-fidelity One Image to 3D Content Generation10 Oct
  2. 2023Large-Vocabulary 3D Diffusion Model with Transformer14 Sep
  3. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  4. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  5. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data13 Jun
  6. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  7. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Images29 Mar
  8. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  9. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  10. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  11. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  12. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
2 more
  1. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models1 Dec
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
21 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  7. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  8. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  9. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  10. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  11. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  12. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  13. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  14. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  15. 2022SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Viewscited by 175
  16. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  17. 2022f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformationcited by 32
  18. 2023TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervisioncited by 28
  19. 2023VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representationcited by 14
  20. 2023Pushing the Limits of 3D Shape Generation at Scalecited by 11
  21. 2023TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fieldscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 11; 2026: 9.

39 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  2. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  3. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  4. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  5. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  6. 2024GVGEN: Text-to-3D Generation with Volumetric Representationcited by 51
  7. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
  8. 2024DiffGS: Functional Gaussian Splatting Diffusioncited by 50
  9. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  10. 2024GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modelingcited by 44
  11. 2024Improved Noise Schedule for Diffusion Trainingcited by 38
  12. 2025Efficient Diffusion Models: A Surveycited by 35
27 more
  1. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  2. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  3. 2025ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generationcited by 30
  4. 2024ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillationcited by 27
  5. 2024GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmarkcited by 11
  6. 20243D Shape Tokenization via Latent Flow Matchingcited by 10
  7. 2024Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusioncited by 9
  8. 2025UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mappingcited by 8
  9. 2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
  10. 2024Simplified Diffusion Schr"odinger Bridgecited by 4
  11. 2025FRIDU: Functional Map Refinement with Guided Image Diffusioncited by 4
  12. 2026GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generationcited by 4
  13. 2024ControLRM: Fast and Controllable 3D Generation via Large Reconstruction Modelcited by 3
  14. 2025Cyc3D: Fine-grained Controllable 3D Generation via Cycle Consistency Regularizationcited by 3
  15. 2026Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformerscited by 3
  16. 2026Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimationcited by 3
  17. 2024SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generationcited by 2
  18. 2026LiTo: Surface Light Field Tokenizationcited by 2
  19. 2026MOC-3D: Manifold-Order Consistency for Text-to-3D Generationcited by 2
  20. 2026ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generationcited by 2
  21. 2025Incorporating Pre-trained Diffusion Models in Solving the Schr"odinger Bridge Problemcited by 1
  22. 2026BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Cloudscited by 1
  23. 2026GaussianGrow: Geometry-aware Gaussian Growing from 3D Point Clouds with Text Guidancecited by 1
  24. 2026GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environmentscited by 1
  25. 2024Several questions of visual generation in 2024cited by 0
  26. 2024Generative AI for Accessible and Inclusive Extended Realitycited by 0
  27. 2025ProcGen3D: Learning Neural Procedural Graph Representations for Image-to-3D Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data
  2. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors
  3. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation
  4. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
  5. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
  6. 2023EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior
  7. 2023PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion
  8. 2023Vox-E: Text-guided Voxel Editing of 3D Objects