mlsanityData to 14 Oct 2026

Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

arXiv 2606.06891 ↗First posted 5 Jun 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 58 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

53 of 58 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Unlocking Dense Metric Depth Estimation in VLMs15 May
  2. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models18 Dec
  3. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning26 Nov
  4. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models22 Jul
  5. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding10 Jul
  6. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  8. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
  9. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  10. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  11. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoning2 Apr
  12. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
5 more
  1. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning1 Mar
  2. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  3. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
  4. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  5. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep

Other references · 36

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  9. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  10. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 692
  11. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  12. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
24 more
  1. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  2. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  3. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  4. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  5. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  6. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  7. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  8. 2025Streaming 4D Visual Geometry Transformercited by 161
  9. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
  10. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 98
  11. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  12. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 82
  13. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 67
  14. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 61
  15. 2026InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streamscited by 56
  16. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  17. 2024Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilitiescited by 48
  18. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  19. 2024StreamChat: Chatting with Streaming Videocited by 24
  20. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 18
  21. 2024CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Mannercited by 11
  22. 20254D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimationcited by 10
  23. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 10
  24. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compressioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MemAct: Streaming Spatial Understanding with Memory and Active Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  2. 2026Geometry-Guided 3D Visual Token Pruning for Video-Language Models
  3. 20263D-Aware VLMs with Implicit and Explicit Geometries
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  5. 2026GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs
  6. 2026SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning
  7. 2026Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
  8. 2026Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding