mlsanityData to 14 Oct 2026

SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE

arXiv 2411.16856 ↗First posted 25 Nov 2024cs.CVcited by 39

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Autoregressive models have demonstrated remarkable success across various fields, from large language models (LLMs) to large multimodal models (LMMs) and 2D content generation, moving closer to artificial general intelligence (AGI). Despite these advances, applying autoregressive approaches to 3D object generation and understanding remains largely unexplored. This paper introduces Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale 3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for efficient autoregressive generation and detailed understanding. By predicting the next scale in a multi-scale latent representation instead of the next single token, SAR3D reduces generation time significantly, achieving fast 3D object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the tokens enriched with hierarchical 3D-aware information, we finetune a pretrained LLM on them, enabling multimodal comprehension of 3D content. Our experiments show that SAR3D surpasses current 3D generation methods in both speed and quality and allows LLMs to interpret and caption 3D models comprehensively.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 84 of its 95 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

84 of 95 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
  2. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 82

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  12. 2023Visual Instruction Tuningcited by 7,869
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  14. 2018Demystifying MMD GANscited by 1,293
  15. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  17. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
  18. 20242D Gaussian Splatting for Geometrically Accurate Radiance Fieldscited by 1,076
  19. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  20. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  21. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  22. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  23. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  24. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  25. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  26. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  27. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  28. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  29. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  30. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  31. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  32. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  33. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  34. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  35. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 521
  36. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 458
  37. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  38. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 19.

39 papers.

Sort:

By area:

  1. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  2. 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generationcited by 30
  3. 2025PAT3D: Physics-Augmented Text-to-3D Scene Generationcited by 16
  4. 2025AR-1-to-3: Single Image to Consistent 3D Object Generation via Next-View Predictioncited by 15
  5. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  6. 20243DEnhancer: Consistent Multi-View Diffusion for 3D Enhancementcited by 12
  7. 2025Are We Ready for RL in Text-to-3D Generation? A Progressive Investigationcited by 10
  8. 2025PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Predictioncited by 9
  9. 2025PhysGen: Physically Grounded 3D Shape Generation for Industrial Designcited by 6
  10. 2025Progressive Supernet Training for Efficient Visual Autoregressive Modelingcited by 5
  11. 2025Aligning Text, Images, and 3D Structure Token-by-Tokencited by 4
  12. 2025GarmentX: Autoregressive Parametric Representations for High-Fidelity 3D Garment Generationcited by 3
27 more
  1. 2025Collaborative Multi-Modal Coding for High-Quality 3D Generationcited by 3
  2. 2025Few-step Flow for 3D Generation via Marginal-Data Transport Distillationcited by 3
  3. 2026CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Modelscited by 3
  4. 2026VAR-3D: View-aware Auto-Regressive Model for Text-to-3D Generation via a 3D Tokenizercited by 3
  5. 2026EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformerscited by 3
  6. 20243D-WAG: Hierarchical Wavelet-Guided Autoregressive Generation for High-Fidelity 3D Shapescited by 2
  7. 2025PacTure: Efficient PBR Texture Generation on Packed Views with Visual Autoregressive Modelscited by 2
  8. 2025AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformerscited by 2
  9. 2025Markovian Scale Prediction: A New Era of Visual Autoregressive Generationcited by 2
  10. 2026TimeMar: Multi-Scale Autoregressive Modeling for Unconditional Time Series Generationcited by 2
  11. 2026Know3D: Prompting 3D Generation with Knowledge from Vision-Language Modelscited by 2
  12. 2026SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestrationcited by 2
  13. 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generationcited by 2
  14. 2026ERGO: Excess-Risk-Guided Optimization for High-Fidelity Monocular 3D Gaussian Splattingcited by 1
  15. 2026SuperVoxelGPT: Adaptive and Ordered 3D Tokenization for Autoregressive Shape Generationcited by 1
  16. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  17. 2025PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Predictioncited by 0
  18. 2025Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstructioncited by 0
  19. 2026PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generationcited by 0
  20. 2026ARGS: Auto-Regressive Gaussian Splatting via Parallel Progressive Next-Scale Predictioncited by 0
  21. 2026VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detectioncited by 0
  22. 2026Invaria: Learning Scale and Density Invariance in Point Clouds via Next-Resolution Predictioncited by 0
  23. 2026MegaParts: Scaling Part-Aware 3D Object Generation to 300 Parts via Token-Efficient Autoregressive Modelingcited by 0
  24. 2026Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformerscited by 0
  25. 2026MeshOctave: Vertex Split-and-Rewire Cascades for Native Mesh Generationcited by 0
  26. 2026SILSA: Sliding-Window Slice Latents for Topology-Preserving High-Resolution 3D Generationcited by 0
  27. 2026Octrees as an Explicit 3D Languagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20243D representation in 512-Byte:Variational tokenizer is the key for autoregressive 3D generation
  2. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction
  3. 2025Recent Advances in 3D Object and Scene Generation: A Survey
  4. 2026Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
  5. 2026Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing
  6. 2023VQ3D: Learning a 3D-Aware Generative Model on ImageNet
  7. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion
  8. 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models