mlsanityData to 14 Oct 2026

Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding

arXiv 2606.19776 ↗First posted 18 Jun 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recently, vision-language models (VLMs) have made significant progress in 3D scene understanding, driving advances in applications such as embodied intelligence and robotic vision. However, existing approaches typically either rely directly on explicit 3D inputs (e.g., point clouds or RGB-D sequences), or introduce an additional 3D geometry encoder to derive 3D-aware visual tokens from 2D images. Such designs structurally decouple 3D geometric perception from the rich 2D semantics learned via vision-language pre-training, hindering the development of a unified 3D vision-language representation. In this work, we propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder. Specifically, Occ-VLM reconstructs 3D scene occupancy as an auxiliary geometric prior, which is utilized to spatially associate foreground 2D tokens with 3D space. These tokens are then decoded by a Large Language Model (LLM) for unified scene understanding. Extensive experiments demonstrate that Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning: it attains state-of-the-art performance on multi-view occupancy prediction, while performing on par with 3D-input VLMs on 3D Visual Question Answering (VQA) and 3D dense captioning benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 60 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

54 of 60 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025LLaVA3: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs20 Nov
  2. 20253D Aware Region Prompted Vision Language Model16 Sep
  3. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  4. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
  5. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  7. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  8. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
  9. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
  10. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 44

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
32 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  4. 2020Convolutional Occupancy Networkscited by 761
  5. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  6. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  7. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  8. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  9. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  10. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  12. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  13. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  14. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  15. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  16. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  17. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  18. 2023Scene as Occupancycited by 209
  19. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  20. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  21. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  22. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  23. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  24. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 157
  25. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  26. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 128
  27. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  28. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  29. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 45
  30. 2025EmbodiedOcc++: Boosting Embodied 3D Occupancy Prediction with Plane Regularization and Uncertainty Samplercited by 27
  31. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  32. 2025SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
  2. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
  3. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
  4. 20263D-Aware VLMs with Implicit and Explicit Geometries
  5. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
  6. 2026VLM3: Vision Language Models Are Native 3D Learners
  7. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
  8. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation