# OctFusion: Octree-based Diffusion Models for 3D Shape Generation

arXiv: https://arxiv.org/abs/2408.14732 · HTML: https://mlsanity.com/paper/2408.14732/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 52 of its 67 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 52)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)

## Built on by (most cited first, 30 of 43)

- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Native and Compact Structured Latents for 3D Generation](https://mlsanity.com/paper/2512.14692/) (2025)
- [Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders](https://mlsanity.com/paper/2412.17808/) (2024)
- [Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attention](https://mlsanity.com/paper/2507.17745/) (2025)
- [Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis](https://mlsanity.com/paper/2507.23785/) (2025)
- [AToken: A Unified Tokenizer for Vision](https://mlsanity.com/paper/2509.14476/) (2025)
- [UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generation](https://mlsanity.com/paper/2509.25079/) (2025)
- [LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion](https://mlsanity.com/paper/2410.01295/) (2024)
- [Efficient Autoregressive Shape Generation via Octree-Based Adaptive Tokenization](https://mlsanity.com/paper/2504.02817/) (2025)
- [Assembler: Scalable 3D Part Assembly via Anchor Point Diffusion](https://mlsanity.com/paper/2506.17074/) (2025)
- [Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings](https://mlsanity.com/paper/2411.08017/) (2024)
- [Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation](https://mlsanity.com/paper/2512.23705/) (2025)
- [NeuroNURBS: Learning Efficient Surface Representations for 3D Solids](https://mlsanity.com/paper/2411.10848/) (2024)
- [DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation](https://mlsanity.com/paper/2412.15200/) (2024)
- [Hyper3D: Efficient 3D Representation via Hybrid Triplane and Octree Feature for Enhanced 3D Shape Variational Auto-Encoders](https://mlsanity.com/paper/2503.10403/) (2025)
- [Representing 3D Shapes With 64 Latent Vectors for 3D Diffusion Models](https://mlsanity.com/paper/2503.08737/) (2025)
- [Nexus: Native Mesh Generation with Diffusion](https://mlsanity.com/paper/2607.13563/) (2026)
- [BiMotion: B-spline Motion for Text-guided Dynamic 3D Character Generation](https://mlsanity.com/paper/2602.18873/) (2026)
- [Constructing a 3D Scene from a Single Image](https://mlsanity.com/paper/2505.15765/) (2025)
- [3D-WAG: Hierarchical Wavelet-Guided Autoregressive Generation for High-Fidelity 3D Shapes](https://mlsanity.com/paper/2411.19037/) (2024)
- [Text-Image Conditioned 3D Generation](https://mlsanity.com/paper/2603.21295/) (2026)
- [TopoMesh: High-Fidelity Mesh Autoencoding via Topological Unification](https://mlsanity.com/paper/2603.24278/) (2026)
- [GenMed: A Pairwise Generative Reformulation of Medical Diagnostic Tasks](https://mlsanity.com/paper/2605.10645/) (2026)
- [WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes](https://mlsanity.com/paper/2605.15843/) (2026)
- [ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation](https://mlsanity.com/paper/2607.06565/) (2026)
- [GALA: Geometry-Aware Local Adaptive Grids for Detailed 3D Generation](https://mlsanity.com/paper/2410.10037/) (2024)
- [MoReMouse: Monocular Reconstruction of Laboratory Mouse](https://mlsanity.com/paper/2507.04258/) (2025)
- [Octree Diffusion for Semantic Scene Generation and Completion](https://mlsanity.com/paper/2509.16483/) (2025)
- [BoxSplitGen: A Generative Model for 3D Part Bounding Boxes in Varying Granularity](https://mlsanity.com/paper/2602.20666/) (2026)
- [MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction](https://mlsanity.com/paper/2607.24436/) (2026)

## Abstract (arXiv)

Diffusion models have emerged as a popular method for 3D generation. However, it is still challenging for diffusion models to efficiently generate diverse and high-quality 3D shapes. In this paper, we introduce OctFusion, which can generate 3D shapes with arbitrary resolutions in 2.5 seconds on a single Nvidia 4090 GPU, and the extracted meshes are guaranteed to be continuous and manifold. The key components of OctFusion are the octree-based latent representation and the accompanying diffusion models. The representation combines the benefits of both implicit neural representations and explicit spatial octrees and is learned with an octree-based variational autoencoder. The proposed diffusion model is a unified multi-scale U-Net that enables weights and computation sharing across different octree levels and avoids the complexity of widely used cascaded diffusion schemes. We verify the effectiveness of OctFusion on the ShapeNet and Objaverse datasets and achieve state-of-the-art performances on shape generation tasks. We demonstrate that OctFusion is extendable and flexible by generating high-quality color fields for textured mesh generation and high-quality 3D shapes conditioned on text prompts, sketches, or category labels. Our code and pre-trained models are available at https://github.com/octree-nn/octfusion.