mlsanityData to 14 Oct 2026

ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

arXiv 2506.01853 ↗First posted 2 Jun 2025cs.CVcited by 38

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni-a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, by performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset. Our work provides an effective attempt at extending multimodal models with basic 3D capabilities, which contributes to future research in 3D-native AI. Project page: https://github.com/JAMESYJL/ShapeLLM-Omni

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 110 of its 126 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

110 of 126 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE25 Nov
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
  3. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
  4. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 106

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  12. 2017Neural Discrete Representation Learningcited by 4,712
38 more
  1. 2024GPT-4o System Cardcited by 4,588
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  10. 2018Demystifying MMD GANscited by 1,293
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  12. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  13. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  14. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  15. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  16. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  17. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  18. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  19. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  20. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  21. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  22. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  23. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  24. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  25. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  26. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  27. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  28. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  29. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  30. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  31. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  32. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 521
  33. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  34. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 458
  35. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  36. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  37. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  38. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 27.

38 papers.

Sort:

By area:

  1. 2025Motus: A Unified Latent Action World Modelcited by 251
  2. 2025PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Imagecited by 37
  3. 2025NANO3D: A Training-Free Approach for Efficient 3D Editing Without Maskscited by 31
  4. 2025From Editor to Dense Geometry Estimatorcited by 24
  5. 2026PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objectscited by 11
  6. 2025Towards Scalable and Consistent 3D Editingcited by 10
  7. 2025Are We Ready for RL in Text-to-3D Generation? A Progressive Investigationcited by 10
  8. 2025CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligencecited by 10
  9. 2026SegviGen: Repurposing 3D Generative Model for Part Segmentationcited by 10
  10. 2026Feedforward 3D Editing Learns from Semantic-Part Transformationcited by 10
  11. 2026ArtLLM: Generating Articulated Assets via 3D LLMcited by 8
  12. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 7
26 more
  1. 2025Feedforward 3D Editing via Text-Steerable Image-to-3Dcited by 7
  2. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  3. 2026GEM: Generative Supervision Helps Embodied Intelligencecited by 6
  4. 2026SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLMcited by 5
  5. 2025Native 3D Editing with Full Attentioncited by 4
  6. 2026LoST: Level of Semantics Tokenization for 3D Shapescited by 4
  7. 2026CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Modelscited by 3
  8. 2026Vinedresser3D: Agentic Text-guided 3D Editingcited by 3
  9. 2026Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generationcited by 3
  10. 2026EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformerscited by 3
  11. 2025Avi: Action from Volumetric Inferencecited by 2
  12. 2026Know3D: Prompting 3D Generation with Knowledge from Vision-Language Modelscited by 2
  13. 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generationcited by 2
  14. 2026Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusioncited by 2
  15. 2026PolyFlow: Continuous Topology Embedding Flow Matching for Artist-style Mesh Generationcited by 1
  16. 2026PhysX-CoT: Structured Physical Reasoning from a Single Image to Simulation-Ready 3D Assetscited by 1
  17. 2026PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generationcited by 0
  18. 2026VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detectioncited by 0
  19. 2026Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editingcited by 0
  20. 2026OmniMech: All-in-one Multimodal Mechanical Benchmark for 3D Reconstructioncited by 0
  21. 2026MegaParts: Scaling Part-Aware 3D Object Generation to 300 Parts via Token-Efficient Autoregressive Modelingcited by 0
  22. 2026Fysiverse-3D-Vision Technical Report: Generating Executable 3D Worlds from Images through Unified Spatial Reasoningcited by 0
  23. 2026Octrees as an Explicit 3D Languagecited by 0
  24. 2026OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generationcited by 0
  25. 2026Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understandingcited by 0
  26. 2026LLM2R: From Large Language Models to 3D Reassemblerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs
  2. 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
  3. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  4. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
  5. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model
  6. 20233D-LLM: Injecting the 3D World into Large Language Models
  7. 2025Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation
  8. 2024AWOL: Analysis WithOut synthesis using Language