# VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder

arXiv: https://arxiv.org/abs/2312.11459 · HTML: https://mlsanity.com/paper/2312.11459/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 47 of its 52 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)

## Built on by (most cited first, 30 of 39)

- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Native and Compact Structured Latents for 3D Generation](https://mlsanity.com/paper/2512.14692/) (2025)
- [3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion](https://mlsanity.com/paper/2409.12957/) (2024)
- [LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation](https://mlsanity.com/paper/2403.12019/) (2024)
- [EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation](https://mlsanity.com/paper/2409.18114/) (2024)
- [GVGEN: Text-to-3D Generation with Volumetric Representation](https://mlsanity.com/paper/2403.12957/) (2024)
- [DiffGS: Functional Gaussian Splatting Diffusion](https://mlsanity.com/paper/2410.19657/) (2024)
- [Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text](https://mlsanity.com/paper/2406.17601/) (2024)
- [Efficient Part-level 3D Object Generation via Dual Volume Packing](https://mlsanity.com/paper/2506.09980/) (2025)
- [GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling](https://mlsanity.com/paper/2403.19655/) (2024)
- [Improved Noise Schedule for Diffusion Training](https://mlsanity.com/paper/2407.03297/) (2024)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis](https://mlsanity.com/paper/2507.23785/) (2025)
- [ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation](https://mlsanity.com/paper/2510.23306/) (2025)
- [View Selection for 3D Captioning via Diffusion Ranking](https://mlsanity.com/paper/2404.07984/) (2024)
- [ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation](https://mlsanity.com/paper/2407.02040/) (2024)
- [GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmark](https://mlsanity.com/paper/2412.09997/) (2024)
- [3D Shape Tokenization via Latent Flow Matching](https://mlsanity.com/paper/2412.15618/) (2024)
- [Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion](https://mlsanity.com/paper/2405.09874/) (2024)
- [UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping](https://mlsanity.com/paper/2502.01846/) (2025)
- [Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data](https://mlsanity.com/paper/2503.21694/) (2025)
- [GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation](https://mlsanity.com/paper/2603.26661/) (2026)
- [Simplified Diffusion Schr"odinger Bridge](https://mlsanity.com/paper/2403.14623/) (2024)
- [FRIDU: Functional Map Refinement with Guided Image Diffusion](https://mlsanity.com/paper/2506.14322/) (2025)
- [Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimation](https://mlsanity.com/paper/2605.03359/) (2026)
- [ControLRM: Fast and Controllable 3D Generation via Large Reconstruction Model](https://mlsanity.com/paper/2410.09592/) (2024)
- [Cyc3D: Fine-grained Controllable 3D Generation via Cycle Consistency Regularization](https://mlsanity.com/paper/2504.14975/) (2025)
- [Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformers](https://mlsanity.com/paper/2604.21592/) (2026)
- [ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation](https://mlsanity.com/paper/2605.21121/) (2026)
- [SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generation](https://mlsanity.com/paper/2410.07658/) (2024)

## Abstract (arXiv)

This paper introduces a pioneering 3D volumetric encoder designed for text-to-3D generation. To scale up the training data for the diffusion model, a lightweight network is developed to efficiently acquire feature volumes from multi-view images. The 3D volumes are then trained on a diffusion model for text-to-3D generation using a 3D U-Net. This research further addresses the challenges of inaccurate object captions and high-dimensional feature volumes. The proposed model, trained on the public Objaverse dataset, demonstrates promising outcomes in producing diverse and recognizable samples from text prompts. Notably, it empowers finer control over object part characteristics through textual cues, fostering model creativity by seamlessly combining multiple concepts within a single object. This research significantly contributes to the progress of 3D generation by introducing an efficient, flexible, and scalable representation methodology.