mlsanityData to 14 Oct 2026

Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling

arXiv 2503.16278 ↗First posted 20 Mar 2025cs.LG cond-mat.mtrl-sci q-bio.BMcited by 8

Not on the map. Why?

Abstract (arXiv)

3D structure modeling is essential across scales, enabling applications from fluid simulation and 3D reconstruction to protein folding and molecular docking. Yet, despite shared 3D spatial patterns, current approaches remain fragmented, with models narrowly specialized for specific domains and unable to generalize across tasks or scales. We propose Uni-3DAR, a unified autoregressive framework for cross-scale 3D generation and understanding. At its core is a coarse-to-fine tokenizer based on octree data structures, which compresses diverse 3D structures into compact 1D token sequences. We further propose a two-level subtree compression strategy, which reduces the octree token sequence by up to 8x. To address the challenge of dynamically varying token positions introduced by compression, we introduce a masked next-token prediction strategy that ensures accurate positional modeling, significantly boosting model performance. Extensive experiments across multiple 3D generation and understanding tasks, including small molecules, proteins, polymers, crystals, and macroscopic 3D objects, validate its effectiveness and versatility. Notably, Uni-3DAR surpasses previous state-of-the-art diffusion models by a substantial margin, achieving up to 256% relative improvement while delivering inference speeds up to 21.8x faster.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 83 of its 122 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

83 of 122 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
38 more
  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  2. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  3. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  4. 2020On Layer Normalization in the Transformer Architecturecited by 922
  5. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  6. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  7. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  8. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  9. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  10. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  11. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  12. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  13. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  14. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  15. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 458
  16. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  17. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 441
  18. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  19. 2022DiffDock: Diffusion Steps, Twists, and Turns for Molecular Dockingcited by 313
  20. 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
  21. 2021Crystal Diffusion Variational Autoencoder for Periodic Material Generationcited by 270
  22. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 269
  23. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  24. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 259
  25. 2020GEOM: Energy-annotated molecular conformations for property prediction and molecular generationcited by 243
  26. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  27. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  28. 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
  29. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 189
  30. 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 186
  31. 2019Symmetry-adapted generation of 3d point sets for the targeted discovery of moleculescited by 172
  32. 2021E(n) Equivariant Normalizing Flowscited by 165
  33. 2022Protein Representation Learning by Geometric Structure Pretrainingcited by 162
  34. 2022A 3D Generative Model for Structure-Based Drug Designcited by 149
  35. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  36. 2022EquiBind: Geometric Deep Learning for Drug Binding Structure Predictioncited by 144
  37. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  38. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 109

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generation
  2. 2025Scalable Autoregressive 3D Molecule Generation
  3. 2026Autoregressive latent diffusion for 3D molecule generation
  4. 2025UniMoMo: Unified Generative Modeling of 3D Molecules for De Novo Binder Design
  5. 2025MODA: A Unified 3D Diffusion Framework for Multi-Task Target-Aware Molecular Generation
  6. 2024Distributional Priors Guided Diffusion for Generating 3D Molecules in Low Data Regimes
  7. 2025Frame-based Equivariant Diffusion Models for 3D Molecular Generation
  8. 20243D representation in 512-Byte:Variational tokenizer is the key for autoregressive 3D generation