mlsanityData to 14 Oct 2026

When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models

arXiv 2405.10255 ↗First posted 16 May 2024cs.CV cs.ROcited by 43

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

As large language models (LLMs) evolve, their integration with 3D spatial data (3D-LLMs) has seen rapid progress, offering unprecedented capabilities for understanding and interacting with physical spaces. This survey provides a comprehensive overview of the methodologies enabling LLMs to process, understand, and generate 3D data. Highlighting the unique advantages of LLMs, such as in-context learning, step-by-step reasoning, open-vocabulary capabilities, and extensive world knowledge, we underscore their potential to significantly advance spatial comprehension and interaction within embodied Artificial Intelligence (AI) systems. Our investigation spans various 3D data representations, from point clouds to Neural Radiance Fields (NeRFs). It examines their integration with LLMs for tasks such as 3D scene understanding, captioning, question-answering, and dialogue, as well as LLM-based agents for spatial reasoning, planning, and navigation. The paper also includes a brief review of other methods that integrate 3D and language. The meta-analysis presented in this paper reveals significant progress yet underscores the necessity for novel approaches to harness the full potential of 3D-LLMs. Hence, with this paper, we aim to chart a course for future research that explores and expands the capabilities of 3D-LLMs in understanding and interacting with the complex 3D world. To support this survey, we have established a project page where papers related to our topic are organized and listed: https://github.com/ActiveVisionLab/Awesome-LLM-3D.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 270 of its 332 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

270 of 332 references.

Closest ancestors · 10

Same direction, newest first

  1. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding6 Jan
  2. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
  3. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding21 Dec
  4. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
  5. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  6. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model29 Nov
  7. 2023Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation6 Aug
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  9. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes4 Jun
  10. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models23 Jul

Other references · 260

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  11. 2023Segment Anythingcited by 8,234
  12. 2023Visual Instruction Tuningcited by 7,869
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  13. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  14. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  15. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  16. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  17. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  18. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  19. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  20. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  21. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  22. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  23. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  24. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  25. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  26. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  27. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  28. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  29. 2023A Survey of Large Language Modelscited by 2,576
  30. 2022Emergent Abilities of Large Language Modelscited by 2,512
  31. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  32. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  33. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  34. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  35. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  36. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  37. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  38. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778

50 most cited of 260 All 260 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 14.

43 papers.

Sort:

By area:

  1. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 49
  2. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  3. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 42
  4. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  5. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 23
  6. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representationcited by 19
  7. 2025Describe Anything Anywhere At Any Momentcited by 19
  8. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  9. 2024Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detectioncited by 17
  10. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  11. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  12. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representationscited by 13
31 more
  1. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  2. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  3. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 9
  4. 2025SpatialLLM: From Multi-modality Data to Urban Spatial Intelligencecited by 8
  5. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  6. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  7. 2025LOSC: LiDAR Open-voc Segmentation Consolidatorcited by 5
  8. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidancecited by 5
  9. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representationscited by 5
  10. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 5
  11. 2024Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenescited by 4
  12. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  13. 2025Aligning Text, Images, and 3D Structure Token-by-Tokencited by 4
  14. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  15. 2025Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptionscited by 3
  16. 2026Vinedresser3D: Agentic Text-guided 3D Editingcited by 3
  17. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Modelscited by 3
  18. 2024Do large language vision models understand 3D shapes?cited by 2
  19. 2025Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actionscited by 1
  20. 2025Error-Driven Scene Editing for 3D Grounding in Large Language Modelscited by 1
  21. 20263DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understandingcited by 1
  22. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understandingcited by 1
  23. 2024Extracting Object Heights From LiDAR & Aerial Imagerycited by 0
  24. 2024Scene Co-pilot: Procedural Text to Video Generation with Human in the Loopcited by 0
  25. 2025DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Taskscited by 0
  26. 2025SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusioncited by 0
  27. 2026Toward Unified Multimodal Representation Learning for Autonomous Drivingcited by 0
  28. 2026OneCanvas: 3D Scene Understanding via Panoramic Reprojectioncited by 0
  29. 2026Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environmentscited by 0
  30. 2026An overview of 3D Vision-Language Modelscited by 0
  31. 2026Octrees as an Explicit 3D Languagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20233D-LLM: Injecting the 3D World into Large Language Models
  2. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning
  3. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Study
  4. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  5. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models
  6. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions
  7. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
  8. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model