mlsanityData to 7 Oct 2026

LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

arXiv 2311.18651 ↗First posted 30 Nov 2023cs.CVcited by 206

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 75 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

58 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 20233D Concept Learning and Reasoning from Multi-View Images20 Mar
  2. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct

Other references · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  8. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  9. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  12. 2022Multimodal Learning with Transformers: A Surveycited by 399
  13. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  14. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  15. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  16. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  18. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  19. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  20. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  21. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  22. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  23. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  24. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  25. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  26. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  27. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  28. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  29. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  30. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  31. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  32. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  33. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  34. 2023CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detectioncited by 51
  35. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  36. 20213D Question Answeringcited by 41
  37. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  38. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 43; 2025: 83; 2026: 79.

206 papers.

Sort:

By area:

  1. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  3. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  4. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  6. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  7. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  8. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  9. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  10. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  11. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  12. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
38 more
  1. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  2. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  3. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  4. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
  5. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  6. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  7. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 63
  8. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  9. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  10. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  11. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  12. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  13. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  14. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  15. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  16. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  17. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  18. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  19. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  20. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  21. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  22. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  23. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  24. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  25. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  26. 2024LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Modelcited by 34
  27. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  28. 2025Detect Anything 3D in the Wildcited by 32
  29. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  30. 20253D Aware Region Prompted Vision Language Modelcited by 31
  31. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worldscited by 29
  32. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  33. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 28
  34. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  35. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  36. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  37. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  38. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24

50 most cited of 206 All 206 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
  2. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  3. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning
  4. 20233D-LLM: Injecting the 3D World into Large Language Models
  5. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
  6. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering
  7. 2025Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation
  8. 20243DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset