mlsanityData to 7 Oct 2026

SpatialLM: Training Large Language Models for Structured Indoor Modeling

arXiv 2506.07491 ↗First posted 9 Jun 2025cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 62 of its 69 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

62 of 69 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  2. 2020Point Transformercited by 1,462
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  8. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  10. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  11. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  13. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  14. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  15. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  16. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  17. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  18. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  19. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  20. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  21. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  22. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  23. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  24. 2024MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priorscited by 198
  25. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  26. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  27. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  28. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  29. 2024MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motioncited by 161
  30. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  31. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  32. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  33. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  34. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 108
  35. 2024SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videoscited by 100
  36. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97
  37. 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
  38. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 42.

63 papers.

Sort:

By area:

  1. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 28
  2. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  3. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  4. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  5. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  6. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 11
  7. 2025OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Modelscited by 9
  8. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  9. 2026ArtLLM: Generating Articulated Assets via 3D LLMcited by 8
  10. 2026Thinking with Spatial Code for Physical-World Video Reasoningcited by 7
  11. 2025Proxy-GS: Unified Occlusion Priors for Training and Inference in Structured 3D Gaussian Splattingcited by 5
  12. 2026Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligencecited by 5
38 more
  1. 2026Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3Dcited by 5
  2. 2026LottieGPT: Tokenizing Vector Animation for Autoregressive Generationcited by 5
  3. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Taskscited by 4
  4. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  5. 2025OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstructioncited by 4
  6. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  7. 2025Zoo3D: Zero-Shot 3D Object Detection at Scene Levelcited by 4
  8. 2025OpenMonoGS-SLAM: Monocular Gaussian Splatting SLAM with Open-set Semanticscited by 4
  9. 2026Repurposing 3D Generative Model for Autoregressive Layout Generationcited by 3
  10. 2025Arcadia: Toward a Full-Lifecycle Framework for Embodied Lifelong Learningcited by 2
  11. 2026OnlineSI: Taming Large Language Model for Online 3D Understanding and Groundingcited by 2
  12. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  13. 2026Few-Shot Incremental 3D Object Detection in Dynamic Indoor Environmentscited by 2
  14. 2026SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generationcited by 2
  15. 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Modelscited by 2
  16. 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generationcited by 2
  17. 2026Towards Spatial Supersensing in the Wildcited by 2
  18. 2025TUN3D: Towards Real-World Scene Understanding from Unposed Imagescited by 1
  19. 2025Fast SceneScript: Fast and Accurate Language-Based 3D Scene Understanding via Multi-Token Predictioncited by 1
  20. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  21. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Modelscited by 1
  22. 2026SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMscited by 1
  23. 20263D Segmentation Using Viewpoint-Dependent Spatial Relationshipscited by 1
  24. 2026GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understandingcited by 1
  25. 2026Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputscited by 1
  26. 2026PartLLM: A Unified Multimodal Foundation for 3D Part Segmentationcited by 1
  27. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  28. 2025Sparse3DPR: Training-Free 3D Hierarchical Scene Parsing and Task-Adaptive Subgraph Reasoning from Sparse RGB Viewscited by 0
  29. 2025DoReMi: Bridging 3D Domains via Topology-Aware Domain-Representation Mixture of Expertscited by 0
  30. 2025DetAny4D: Detect Anything 4D Temporally in a Streaming RGB Videocited by 0
  31. 2026GeoSense: Internalizing Geometric Necessity Perception for Multimodal Reasoningcited by 0
  32. 2026SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenescited by 0
  33. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 0
  34. 2026Lifting Unlabeled Internet-level Data for 3D Scene Understandingcited by 0
  35. 2026Coarse Semantic Injection for LLM-Conditioned Structured Indoor Predictioncited by 0
  36. 2026Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Modelscited by 0
  37. 2026Vector Map as Language: Toward Unified Remote Sensing Vector Mappingcited by 0
  38. 2026SemanticXR: Low Power and Real-time Queryable Semantic Mapping with an Object-Level Device-Cloud Architecturecited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  2. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
  3. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs
  4. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  5. 20243DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset
  6. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibility
  7. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generation
  8. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations