mlsanityData to 14 Oct 2026

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding

arXiv 2312.14074 ↗First posted 21 Dec 2023cs.CVcited by 95

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recently, Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have shown promise in instruction following and 2D image understanding. While these models are powerful, they have not yet been developed to comprehend the more challenging 3D physical scenes, especially when it comes to the sparse outdoor LiDAR data. In this paper, we introduce LiDAR-LLM, which takes raw LiDAR data as input and harnesses the remarkable reasoning capabilities of LLMs to gain a comprehensive understanding of outdoor 3D scenes. The central insight of our LiDAR-LLM is the reformulation of 3D outdoor scene cognition as a language modeling problem, encompassing tasks such as 3D captioning, 3D grounding, 3D question answering, etc. Specifically, due to the scarcity of 3D LiDAR-text pairing data, we introduce a three-stage training strategy and generate relevant datasets, progressively aligning the 3D modality with the language embedding space of LLM. Furthermore, we design a View-Aware Transformer (VAT) to connect the 3D encoder with the LLM, which effectively bridges the modality gap and enhances the LLM's spatial orientation comprehension of visual features. Our experiments show that LiDAR-LLM possesses favorable capabilities to comprehend various instructions regarding 3D scenes and engage in complex spatial reasoning. LiDAR-LLM attains a 40.9 BLEU-1 on the 3D captioning task and achieves a 63.1% classification accuracy and a 14.3% BEV mIoU on the 3D grounding task. Web page: https://sites.google.com/view/lidar-llm

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 53 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

43 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes12 Apr

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2015VQA: Visual Question Answeringcited by 2,506
30 more
  1. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  5. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  6. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  7. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  8. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  9. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  10. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  11. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  12. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  13. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  14. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 128
  15. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  16. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  17. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  18. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  19. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  20. 2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
  21. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  22. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  23. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  24. 20183D-Aware Scene Manipulation via Inverse Graphicscited by 24
  25. 2022BEV-SAN: Accurate BEV 3D Object Detection via Slice Attention Networkscited by 20
  26. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  27. 2022Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-worldcited by 17
  28. 20223D Concept Grounding on Neural Fieldscited by 13
  29. 2023ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selectioncited by 7
  30. 2023Unlock the Power: Competitive Distillation for Multi-Modal Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 46; 2026: 20.

95 papers.

Sort:

By area:

  1. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 184
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 176
  3. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  4. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  5. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 103
  6. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  7. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 83
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  9. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 66
  10. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  11. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  12. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
38 more
  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  2. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 46
  3. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  4. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 43
  5. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 42
  6. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  7. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 38
  8. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  10. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  11. 2024Diving into Self-Evolving Training for Multimodal Reasoningcited by 32
  12. 2024Unified Language-driven Zero-shot Domain Adaptationcited by 30
  13. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  14. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  15. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  16. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  17. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  18. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 21
  19. 2025URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Modelcited by 21
  20. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  21. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 20
  22. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 18
  23. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  24. 2024CSA: Data-efficient Mapping of Unimodal Features to Multimodal Featurescited by 12
  25. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMscited by 12
  26. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  27. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  28. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 9
  29. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  30. 2025StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusioncited by 7
  31. 2025RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequencecited by 7
  32. 2025CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learningcited by 7
  33. 2025CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulationcited by 7
  34. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compressioncited by 7
  35. 2025CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Modelcited by 7
  36. 2024Typicalness-Aware Learning for Failure Detectioncited by 6
  37. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  38. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20233D-LLM: Injecting the 3D World into Large Language Models
  2. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  3. 20263DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  5. 2024Language-Image Models with 3D Understanding
  6. 2024Grounded 3D-LLM with Referent Tokens
  7. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  8. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors