# OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generation

arXiv: https://arxiv.org/abs/2504.09975 · HTML: https://mlsanity.com/paper/2504.09975/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 64 of its 85 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [3D Neural Field Generation using Triplane Diffusion](https://mlsanity.com/paper/2211.16677/) (2022)

## Built on by (most cited first, 30 of 30)

- [Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assets](https://mlsanity.com/paper/2505.07747/) (2025)
- [VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Space](https://mlsanity.com/paper/2508.19247/) (2025)
- [NANO3D: A Training-Free Approach for Efficient 3D Editing Without Masks](https://mlsanity.com/paper/2510.15019/) (2025)
- [CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence](https://mlsanity.com/paper/2512.12768/) (2025)
- [SegviGen: Repurposing 3D Generative Model for Part Segmentation](https://mlsanity.com/paper/2603.16869/) (2026)
- [MVPainter: Accurate and Detailed 3D Texture Generation via Multi-View Diffusion with Geometric Control](https://mlsanity.com/paper/2505.12635/) (2025)
- [FACE: A Face-based Autoregressive Representation for High-Fidelity and Efficient Mesh Generation](https://mlsanity.com/paper/2603.01515/) (2026)
- [Nexus: Native Mesh Generation with Diffusion](https://mlsanity.com/paper/2607.13563/) (2026)
- [LoST: Level of Semantics Tokenization for 3D Shapes](https://mlsanity.com/paper/2603.17995/) (2026)
- [MeshTailor: Cutting Seams via Generative Mesh Traversal](https://mlsanity.com/paper/2603.27309/) (2026)
- [VAR-3D: View-aware Auto-Regressive Model for Text-to-3D Generation via a 3D Tokenizer](https://mlsanity.com/paper/2602.13818/) (2026)
- [Repurposing 3D Generative Model for Autoregressive Layout Generation](https://mlsanity.com/paper/2604.16299/) (2026)
- [Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling](https://mlsanity.com/paper/2510.23605/) (2025)
- [ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation](https://mlsanity.com/paper/2607.06565/) (2026)
- [Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion](https://mlsanity.com/paper/2608.19567/) (2026)
- [SuperVoxelGPT: Adaptive and Ordered 3D Tokenization for Autoregressive Shape Generation](https://mlsanity.com/paper/2605.29655/) (2026)
- [HairLRM: Strand-based Hair Modeling via Large Reconstruction Models](https://mlsanity.com/paper/2606.15238/) (2026)
- [Adaptive Volumetric Mechanical Property Fields Invariant to Resolution](https://mlsanity.com/paper/2606.18231/) (2026)
- [EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learning](https://mlsanity.com/paper/2607.07187/) (2026)
- [aDSL: Agentic 3D Creation via Joint Agent-Program Design](https://mlsanity.com/paper/2608.17975/) (2026)
- [A Few-Step Generative Model on Cumulative Flow Maps](https://mlsanity.com/paper/2605.03623/) (2026)
- [Quantum-Enhanced Similarity Measures for Polarimetric Materials Classification](https://mlsanity.com/paper/2606.07766/) (2026)
- [MegaParts: Scaling Part-Aware 3D Object Generation to 300 Parts via Token-Efficient Autoregressive Modeling](https://mlsanity.com/paper/2608.14783/) (2026)
- [S2GS: Structured Sparse Gaussian Streaming for Efficient Free-Viewpoint Video Reconstruction on Edge-IoT Devices](https://mlsanity.com/paper/2608.19639/) (2026)
- [ZODIAC: Zero-shot Octree-based Diffusion for Anatomical Completion](https://mlsanity.com/paper/2608.24422/) (2026)
- [OrnaStyler: Ornament-Aware Latent Editing for Content-Preserving 3D Stylization](https://mlsanity.com/paper/2608.29905/) (2026)
- [MeshOctave: Vertex Split-and-Rewire Cascades for Native Mesh Generation](https://mlsanity.com/paper/2609.38985/) (2026)
- [Octrees as an Explicit 3D Language](https://mlsanity.com/paper/2610.02388/) (2026)
- [OctMole: Multiscale 3D Molecular Generation with Hierarchical Spatial Flows](https://mlsanity.com/paper/2610.16586/) (2026)
- [Q-MAGIC: Quad-only Mesh Autoregressive Generation via Singularity-Aware Circular Relative Offset Tokenization](https://mlsanity.com/paper/2610.19777/) (2026)

## Abstract (arXiv)

Autoregressive models have achieved remarkable success across various domains, yet their performance in 3D shape generation lags significantly behind that of diffusion models. In this paper, we introduce OctGPT, a novel multiscale autoregressive model for 3D shape generation that dramatically improves the efficiency and performance of prior 3D autoregressive approaches, while rivaling or surpassing state-of-the-art diffusion models. Our method employs a serialized octree representation to efficiently capture the hierarchical and spatial structures of 3D shapes. Coarse geometry is encoded via octree structures, while fine-grained details are represented by binary tokens generated using a vector quantized variational autoencoder (VQVAE), transforming 3D shapes into compact multiscale binary sequences suitable for autoregressive prediction. To address the computational challenges of handling long sequences, we incorporate octree-based transformers enhanced with 3D rotary positional encodings, scale-specific embeddings, and token-parallel generation schemes. These innovations reduce training time by 13 folds and generation time by 69 folds, enabling the efficient training of high-resolution 3D shapes, e.g.,10243, on just four NVIDIA 4090 GPUs only within days. OctGPT showcases exceptional versatility across various tasks, including text-, sketch-, and image-conditioned generation, as well as scene-level synthesis involving multiple objects. Extensive experiments demonstrate that OctGPT accelerates convergence and improves generation quality over prior autoregressive methods, offering a new paradigm for high-quality, scalable 3D content creation. Our code and trained models are available at https://github.com/octree-nn/octgpt.