mlsanityData to 7 Oct 2026

Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following

arXiv 2309.00615 ↗First posted 1 Sep 2023cs.CV cs.AI cs.CLcited by 192

Not on the map. Why?

Cited by 74 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. Guided by ImageBind, we construct a joint embedding space between 3D and multi-modalities, enabling many promising applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D open-world understanding. On top of this, we further present Point-LLM, the first 3D large language model (LLM) following 3D multi-modal instructions. By parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi-modal question-answering capacity. We hope our work may cast a light on the community for extending 3D point clouds to multi-modality applications. Code is available at https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Builds on 80 of its 99 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

80 of 99 references.

References · 80

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2016Layer Normalizationcited by 5,579
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  13. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  14. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  17. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  18. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  19. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  20. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  21. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  23. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  24. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  25. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  26. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  27. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  28. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  29. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  30. 2021Group-Free 3D Object Detection via Transformerscited by 215
  31. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  32. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  33. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  34. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  35. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  36. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  37. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  38. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 70; 2025: 71; 2026: 28.

192 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  6. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  7. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  8. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  9. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  11. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  12. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 140
38 more
  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  3. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  4. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  5. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  7. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  8. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  9. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  10. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  11. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  12. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  13. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  14. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  15. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  16. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  17. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 74
  18. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  19. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  20. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  21. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 61
  22. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  23. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  24. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  25. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  26. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  27. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  28. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  29. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  30. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  31. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  32. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  33. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 38
  34. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  35. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  36. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 38
  37. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedbackcited by 36
  38. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36

50 most cited of 192 All 192 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  2. 20233D-LLM: Injecting the 3D World into Large Language Models
  3. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  4. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Study
  5. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  6. 2026PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought
  7. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
  8. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Model