mlsanityData to 7 Oct 2026

Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

arXiv 2403.11401 ↗First posted 18 Mar 2024cs.CV cs.AIcited by 170

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

This paper introduces Scene-LLM, a 3D-visual-language model that enhances embodied agents' abilities in interactive 3D indoor environments by integrating the reasoning strengths of Large Language Models (LLMs). Scene-LLM adopts a hybrid 3D visual feature representation, that incorporates dense spatial information and supports scene state updates. The model employs a projection layer to efficiently project these features in the pre-trained textual embedding space, enabling effective interpretation of 3D visual information. Unique to our approach is the integration of both scene-level and ego-centric 3D information. This combination is pivotal for interactive planning, where scene-level data supports global planning and ego-centric data is important for localization. Notably, we use ego-centric 3D frame features for feature alignment, an efficient technique that enhances the model's ability to align features of small objects within the scene. Our experiments with Scene-LLM demonstrate its strong capabilities in dense captioning, question answering, and interactive planning. We believe Scene-LLM advances the field of 3D visual understanding and reasoning, offering new possibilities for sophisticated agent interactions in indoor settings.

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 77 of its 87 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

77 of 87 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning12 May
  2. 20233D Concept Learning and Reasoning from Multi-View Images20 Mar
  3. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct
  4. 2022JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents28 Aug

Other references · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
38 more
  1. 2020Point Transformercited by 1,462
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  7. 2022A Generalist Agentcited by 802
  8. 2023Large Language Modelscited by 774
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  13. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  14. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  15. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  16. 2023LERF: Language Embedded Radiance Fieldscited by 538
  17. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  19. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  20. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  21. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  22. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  23. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  24. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  25. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  26. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  27. 2022Visual Language Maps for Robot Navigationcited by 362
  28. 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
  29. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  30. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  31. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  32. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  33. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  34. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  35. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  36. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  37. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  38. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 91; 2026: 49.

170 papers.

Sort:

By area:

  1. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  3. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  4. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  5. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  6. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  7. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  8. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  9. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  10. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  11. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  12. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
38 more
  1. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  2. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  3. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  4. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  5. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  6. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  7. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  10. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  11. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  12. 2024LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Modelcited by 34
  13. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  14. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  15. 20253D Aware Region Prompted Vision Language Modelcited by 31
  16. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  17. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  18. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Taskscited by 27
  19. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  20. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  21. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  22. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  23. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  24. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  25. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  26. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  27. 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometricscited by 21
  28. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  29. 2024Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLMcited by 20
  30. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 19
  31. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  32. 2025MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Meshcited by 18
  33. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
  34. 2025AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Modelscited by 15
  35. 2025Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotationscited by 14
  36. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  37. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  38. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11

50 most cited of 170 All 170 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
  2. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understanding
  3. 2025A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding
  4. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
  5. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  6. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
  7. 2024Think-Program-reCtify: 3D Situated Reasoning with Large Language Models
  8. 20233D-LLM: Injecting the 3D World into Large Language Models