Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling
Not on the map. Why?
Abstract (arXiv)
3D structure modeling is essential across scales, enabling applications from fluid simulation and 3D reconstruction to protein folding and molecular docking. Yet, despite shared 3D spatial patterns, current approaches remain fragmented, with models narrowly specialized for specific domains and unable to generalize across tasks or scales. We propose Uni-3DAR, a unified autoregressive framework for cross-scale 3D generation and understanding. At its core is a coarse-to-fine tokenizer based on octree data structures, which compresses diverse 3D structures into compact 1D token sequences. We further propose a two-level subtree compression strategy, which reduces the octree token sequence by up to 8x. To address the challenge of dynamically varying token positions introduced by compression, we introduce a masked next-token prediction strategy that ensures accurate positional modeling, significantly boosting model performance. Extensive experiments across multiple 3D generation and understanding tasks, including small molecules, proteins, polymers, crystals, and macroscopic 3D objects, validate its effectiveness and versatility. Notably, Uni-3DAR surpasses previous state-of-the-art diffusion models by a substantial margin, achieving up to 256% relative improvement while delivering inference speeds up to 21.8x faster.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 83 of its 122 references that are in the map. First posted on arXiv 20 Mar 2025.
Builds on
83 of 122 references.
References · 83
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2017Neural Discrete Representation Learningcited by 4,712
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
- 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
- 2020GLU Variants Improve Transformercited by 1,804
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
- 2019Root Mean Square Layer Normalizationcited by 1,480
38 more
- 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
- 2022MaskGIT: Masked Generative Image Transformercited by 1,099
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
- 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
- 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
- 2024Autoregressive Image Generation without Vector Quantizationcited by 651
- 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
- 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
- 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
- 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
- 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
- 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
- 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 458
- 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
- 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 441
- 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
- 2022DiffDock: Diffusion Steps, Twists, and Turns for Molecular Dockingcited by 313
- 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
- 2021Crystal Diffusion Variational Autoencoder for Periodic Material Generationcited by 270
- 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 269
- 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
- 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 259
- 2020GEOM: Energy-annotated molecular conformations for property prediction and molecular generationcited by 243
- 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
- 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
- 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
- 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 189
- 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 186
- 2019Symmetry-adapted generation of 3d point sets for the targeted discovery of moleculescited by 172
- 2021E(n) Equivariant Normalizing Flowscited by 165
- 2022Protein Representation Learning by Geometric Structure Pretrainingcited by 162
- 2022A 3D Generative Model for Structure-Based Drug Designcited by 149
- 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
- 2022EquiBind: Geometric Deep Learning for Drug Binding Structure Predictioncited by 144
- 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
- 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 109
50 most cited of 83 All 83 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 6.
8 papers.
- 2025Hyperspherical Latents Improve Continuous-Token Autoregressive Generationcited by 16
- 2026Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materialscited by 5
- 2025NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generationcited by 3
- 2026DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoisingcited by 2
- 2026SinAE: A Single-Architecture Flow-Matching Autoencoder for Cross-Domain Atomic Systemscited by 1
- 2026Octrees as an Explicit 3D Languagecited by 0
- 2026A self-learning scientific agent for X-ray diffractioncited by 0
- 2026OctMole: Multiscale 3D Molecular Generation with Hierarchical Spatial Flowscited by 0
- 2026OctMole: Multiscale 3D Molecular Generation with Hierarchical Spatial Flows12 Oct 2026
- 2026A self-learning scientific agent for X-ray diffraction6 Oct 2026
- 2026Octrees as an Explicit 3D Language1 Oct 2026
- 2026SinAE: A Single-Architecture Flow-Matching Autoencoder for Cross-Domain Atomic Systems14 Jul 2026
- 2026DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising19 Mar 2026
- 2026Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materials24 Feb 2026
- 2025NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generation5 Dec 2025
- 2025Hyperspherical Latents Improve Continuous-Token Autoregressive Generation29 Sep 2025
Similar, not cited
Close in wording, no citation either way
- 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generation
- 2025Scalable Autoregressive 3D Molecule Generation
- 2026Autoregressive latent diffusion for 3D molecule generation
- 2025UniMoMo: Unified Generative Modeling of 3D Molecules for De Novo Binder Design
- 2025MODA: A Unified 3D Diffusion Framework for Multi-Task Target-Aware Molecular Generation
- 2024Distributional Priors Guided Diffusion for Generating 3D Molecules in Low Data Regimes
- 2025Frame-based Equivariant Diffusion Models for 3D Molecular Generation
- 20243D representation in 512-Byte:Variational tokenizer is the key for autoregressive 3D generation