# ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

arXiv: https://arxiv.org/abs/2506.01853 · HTML: https://mlsanity.com/paper/2506.01853/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 110 of its 126 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 110)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)

## Built on by (most cited first, 30 of 38)

- [Motus: A Unified Latent Action World Model](https://mlsanity.com/paper/2512.13030/) (2025)
- [PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image](https://mlsanity.com/paper/2511.13648/) (2025)
- [NANO3D: A Training-Free Approach for Efficient 3D Editing Without Masks](https://mlsanity.com/paper/2510.15019/) (2025)
- [From Editor to Dense Geometry Estimator](https://mlsanity.com/paper/2509.04338/) (2025)
- [PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects](https://mlsanity.com/paper/2605.21572/) (2026)
- [Towards Scalable and Consistent 3D Editing](https://mlsanity.com/paper/2510.02994/) (2025)
- [Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation](https://mlsanity.com/paper/2512.10949/) (2025)
- [CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence](https://mlsanity.com/paper/2512.12768/) (2025)
- [SegviGen: Repurposing 3D Generative Model for Part Segmentation](https://mlsanity.com/paper/2603.16869/) (2026)
- [Feedforward 3D Editing Learns from Semantic-Part Transformation](https://mlsanity.com/paper/2605.27351/) (2026)
- [ArtLLM: Generating Articulated Assets via 3D LLM](https://mlsanity.com/paper/2603.01142/) (2026)
- [Part-X-MLLM: Part-aware 3D Multimodal Large Language Model](https://mlsanity.com/paper/2511.13647/) (2025)
- [Feedforward 3D Editing via Text-Steerable Image-to-3D](https://mlsanity.com/paper/2512.13678/) (2025)
- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [GEM: Generative Supervision Helps Embodied Intelligence](https://mlsanity.com/paper/2605.28548/) (2026)
- [SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM](https://mlsanity.com/paper/2603.23386/) (2026)
- [Native 3D Editing with Full Attention](https://mlsanity.com/paper/2511.17501/) (2025)
- [LoST: Level of Semantics Tokenization for 3D Shapes](https://mlsanity.com/paper/2603.17995/) (2026)
- [CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models](https://mlsanity.com/paper/2601.21798/) (2026)
- [Vinedresser3D: Agentic Text-guided 3D Editing](https://mlsanity.com/paper/2602.19542/) (2026)
- [Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation](https://mlsanity.com/paper/2604.02289/) (2026)
- [EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers](https://mlsanity.com/paper/2605.16745/) (2026)
- [Avi: Action from Volumetric Inference](https://mlsanity.com/paper/2510.21746/) (2025)
- [Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models](https://mlsanity.com/paper/2603.22782/) (2026)
- [ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation](https://mlsanity.com/paper/2607.06565/) (2026)
- [Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion](https://mlsanity.com/paper/2608.19567/) (2026)
- [PolyFlow: Continuous Topology Embedding Flow Matching for Artist-style Mesh Generation](https://mlsanity.com/paper/2606.30673/) (2026)
- [PhysX-CoT: Structured Physical Reasoning from a Single Image to Simulation-Ready 3D Assets](https://mlsanity.com/paper/2608.08053/) (2026)
- [PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generation](https://mlsanity.com/paper/2602.03533/) (2026)
- [VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detection](https://mlsanity.com/paper/2605.01365/) (2026)

## Abstract (arXiv)

Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni-a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, by performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset. Our work provides an effective attempt at extending multimodal models with basic 3D capabilities, which contributes to future research in 3D-native AI. Project page: https://github.com/JAMESYJL/ShapeLLM-Omni