mlsanityData to 14 Oct 2026

LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences

arXiv 2412.01292 ↗First posted 2 Dec 2024cs.CVcited by 38

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing attention, which is crucial for developing embodied AI within 3D scenes, such as visual navigation and embodied question answering. Due to the high density of visual features, especially in large 3D scenes, accurately locating task-relevant visual information is challenging. Existing works attempt to segment all objects and consider their features as scene representations. However, these task-agnostic object features include much redundant information and missing details for the task-relevant area. To tackle these problems, we propose LSceneLLM, an adaptive framework that automatically identifies task-relevant areas by leveraging LLM's visual preference for different tasks, followed by a plug-and-play scene magnifier module to capture fine-grained details in focused areas. Specifically, a dense token selector examines the attention map of LLM to identify visual preferences for the instruction input. It then magnifies fine-grained details of the focusing area. An adaptive self-attention module is leveraged to fuse the coarse-grained and selected fine-grained visual information. To comprehensively evaluate the large scene understanding ability of 3D-VLMs, we further introduce a cross-room understanding benchmark, XR-Scene, which contains a series of large scene understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption. Experiments show that our method surpasses existing methods on both large scene understanding and existing scene understanding benchmarks. Plunging our scene magnifier module into the existing 3D-VLMs also brings significant improvement.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

35 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  2. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding21 Dec
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 32

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  11. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  12. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
20 more
  1. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  2. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  3. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  4. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  5. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  6. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  7. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  8. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  9. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  10. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  11. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  12. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  13. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  14. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  15. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 83
  16. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  17. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  18. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 62
  19. 2023Uni3D: A Unified Baseline for Multi-dataset 3D Object Detectioncited by 39
  20. 2023FGPrompt: Fine-grained Goal Prompting for Image-goal Navigationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 19.

38 papers.

Sort:

By area:

  1. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  2. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
  3. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understandingcited by 47
  4. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  5. 2025Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentationcited by 12
  6. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  7. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMscited by 8
  8. 2025SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Modelscited by 7
  9. 2025Understanding Dynamic Scenes in Ego Centric 4D Point Cloudscited by 6
  10. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  11. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibilitycited by 5
  12. 2025Zoo3D: Zero-Shot 3D Object Detection at Scene Levelcited by 4
26 more
  1. 20263D Instruction Ambiguity Detectioncited by 3
  2. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 3
  3. 2025Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Modelcited by 2
  4. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  5. 2026C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoningcited by 2
  6. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  7. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  8. 2026Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLMcited by 2
  9. 2025TUN3D: Towards Real-World Scene Understanding from Unposed Imagescited by 1
  10. 2025Error-Driven Scene Editing for 3D Grounding in Large Language Modelscited by 1
  11. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understandingcited by 1
  12. 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Modelscited by 1
  13. 2026Efficient Encoder-Free Fourier-based 3D Large Multimodal Modelcited by 1
  14. 2026VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoningcited by 1
  15. 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoningcited by 1
  16. 2026D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Modelscited by 1
  17. 2025VoxRep: Enhancing 3D Spatial Understanding in 2D Vision-Language Models via Voxel Representationcited by 0
  18. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  19. 2026Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Predictioncited by 0
  20. 2026Agentic Collaborative Cognition for Zero-Shot 3D Understandingcited by 0
  21. 2026CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Modelscited by 0
  22. 2026SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Expertscited by 0
  23. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0
  24. 2026Task-Adaptive Grounded 3D-Programmers Using 2D VLMscited by 0
  25. 2026EgoAlign: Situated 3D Understanding without Situation Descriptionscited by 0
  26. 2026LLM2R: From Large Language Models to 3D Reassemblerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
  2. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
  3. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  4. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  5. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  6. 2025Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
  7. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  8. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding