mlsanityData to 14 Oct 2026

PointLLM: Empowering Large Language Models to Understand Point Clouds

arXiv 2308.16911 ↗First posted 31 Aug 2023cs.CV cs.AI cs.CLcited by 333

Not on the map. Why?

Cited by 138 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The unprecedented advancements in Large Language Models (LLMs) have shown a profound impact on natural language processing but are yet to fully embrace the realm of 3D understanding. This paper introduces PointLLM, a preliminary effort to fill this gap, enabling LLMs to understand point clouds and offering a new avenue beyond 2D visual data. PointLLM understands colored object point clouds with human instructions and generates contextually appropriate responses, illustrating its grasp of point clouds and common sense. Specifically, it leverages a point cloud encoder with a powerful LLM to effectively fuse geometric, appearance, and linguistic information. We collect a novel dataset comprising 660K simple and 70K complex point-text instruction pairs to enable a two-stage training strategy: aligning latent spaces and subsequently instruction-tuning the unified model. To rigorously evaluate the perceptual and generalization capabilities of PointLLM, we establish two benchmarks: Generative 3D Object Classification and 3D Object Captioning, assessed through three different methods, including human evaluation, GPT-4/ChatGPT evaluation, and traditional metrics. Experimental results reveal PointLLM's superior performance over existing 2D and 3D baselines, with a notable achievement in human-evaluated object captioning tasks where it surpasses human annotators in over 50% of the samples. Codes, datasets, and benchmarks are available at https://github.com/OpenRobotLab/PointLLM .

Key facts. Cited by 333 papers in the map (data through 14 Oct 2026). Builds on 60 of its 75 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

60 of 75 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  10. 2023Segment Anythingcited by 8,234
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  14. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  15. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  16. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  17. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  18. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  19. 2023A Survey on Multimodal Large Language Modelscited by 909
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  21. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  22. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  23. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  24. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  25. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  26. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  27. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  28. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  29. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  30. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  31. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  32. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  33. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  34. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  35. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  36. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  37. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  38. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 83; 2025: 113; 2026: 115.

333 papers.

Sort:

By area:

  1. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  4. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  10. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  11. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  12. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 184
38 more
  1. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  2. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  3. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 144
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  5. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  6. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 125
  8. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  9. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  10. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  11. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  12. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  13. 2023GraphLLM: Boosting Graph Reasoning Ability of Large Language Modelcited by 107
  14. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  15. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  16. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 100
  17. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  18. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  19. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 89
  20. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  21. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  22. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  23. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  24. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  25. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 66
  26. 2024Empowering LLMs to Understand and Generate Complex Vector Graphicscited by 66
  27. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 66
  28. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  29. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 62
  30. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  31. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  32. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 49
  33. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 47
  34. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  35. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  36. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  37. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  38. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43

50 most cited of 333 All 333 (JSON)