mlsanityData to 7 Oct 2026

From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models

arXiv 2607.03661 ↗First posted 4 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Point cloud-based understanding has become an important enabler for facility operation and maintenance involving indoor building components. However, existing methods output only discrete labels without explaining component functions or natural language interactions. This paper proposes Building-MLLM, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks. Building-MLLM addresses semantic concentration through four domain-specific mechanisms: Point Information Enhancer for task-relevant semantics, Geometry-Preserving Regularization preventing geometric erosion, fixed textual prefix for domain stabilization, and multi-dimensional LoRA balancing recognition with reasoning. A multi-constraint progressive instruction-generation engine is developed to compile a synthetic point cloud-text dataset with 4198 objects, 37,782 instruction-following pairs, and 47 categories. Experiments show that Building-MLLM achieves 88.00%, 65.10%, and 68.14% on the three task types, respectively, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 57 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

25 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
  3. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
  4. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  12. 2020Point Transformercited by 1,462
9 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  3. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  4. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  5. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  6. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  7. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  8. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  9. 2021IFCNet: A Benchmark Dataset for IFC Entity Classificationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SpatialLM: Training Large Language Models for Structured Indoor Modeling
  2. 2026Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning
  3. 2026Tokenization Allows Multimodal Large Language Models to Understand, Generate and Edit Architectural Floor Plans
  4. 2026DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings
  5. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs
  6. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations
  7. 2026IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning
  8. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation