mlsanityData to 7 Oct 2026

ShapeLLM: Universal 3D Object Understanding for Embodied Interaction

arXiv 2402.17766 ↗First posted 27 Feb 2024cs.CVcited by 133

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 162 of its 208 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

162 of 208 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
  3. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  5. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 157

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  13. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  14. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  15. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  18. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  19. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  20. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  21. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  22. 2019Language Models as Knowledge Bases?cited by 1,922
  23. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  24. 2022Visual Prompt Tuningcited by 1,674
  25. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  26. 2020Point Transformercited by 1,462
  27. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  28. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  29. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  30. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  31. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  32. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  33. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  34. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  35. 2019How Can We Know What Language Models Know?cited by 953
  36. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  37. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  38. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786

50 most cited of 157 All 157 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 61; 2026: 47.

133 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  4. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  5. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  6. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  7. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  8. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 52
  9. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  10. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  11. 2025X-Part: high fidelity and structure coherent shape decompositioncited by 42
  12. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
38 more
  1. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  2. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 38
  3. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  4. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  5. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  6. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worldscited by 29
  7. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  8. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  9. 2025PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understandingcited by 24
  10. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  11. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  12. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  13. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 19
  14. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  15. 2024SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Modelcited by 18
  16. 2025Reasoning in Space via Grounding in the Worldcited by 18
  17. 2025OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Modelcited by 17
  18. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
  19. 2024Positional Prompt Tuning for Efficient 3D Representation Learningcited by 15
  20. 2024RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learningcited by 15
  21. 2024PointNet with KAN versus PointNet with MLP for 3D Classification and Segmentation of Point Setscited by 15
  22. 2025MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decodingcited by 14
  23. 2024A Survey on Human-Centric LLMscited by 13
  24. 2025PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learningcited by 13
  25. 2025GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Modelcited by 13
  26. 2024Duoduo CLIP: Efficient 3D Understanding with Multi-View Imagescited by 12
  27. 20253DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scalecited by 12
  28. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  29. 2025MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skillscited by 10
  30. 2024Kestrel: 3D Multimodal LLM for Part-Aware Grounded Descriptioncited by 9
  31. 2024More Text, Less Point: Towards 3D Data-Efficient Point-Language Understandingcited by 9
  32. 2025Just Noticeable Difference for Large Multimodal Modelscited by 9
  33. 2025CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligencecited by 9
  34. 2024PerLA: Perceptive 3D Language Assistantcited by 8
  35. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  36. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  37. 2025Are We Ready for RL in Text-to-3D Generation? A Progressive Investigationcited by 8
  38. 2026ArtLLM: Generating Articulated Assets via 3D LLMcited by 8

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
  4. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  5. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models
  6. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
  7. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning
  8. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding