mlsanityData to 7 Oct 2026

Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding

arXiv 2511.21191 ↗First posted 26 Nov 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advances in 3D vision-language models (VLMs) highlight a strong potential for 3D scene understanding and reasoning. However, effectively tokenizing 3D scenes into holistic scene tokens, and leveraging these tokens across diverse 3D understanding tasks, remain highly challenging. We present NDTokenizer3D, a generalist 3D VLM that performs a wide range of 3D scene understanding tasks while naturally supporting human interactions, thereby bridging language-level reasoning with 3D spatial understanding. The core of our approach is a novel three-stage scene tokenization pipeline built upon a Multi-Scale Normal Distributions Transform (NDT) representation, paired with a Multi-Scale NDT Decoder (MSDec). Specifically, NDTokenizer3D first constructs a multi-scale NDT representation from raw high-resolution point clouds, preserving both global context and fine-grained geometric details. Next, the MSDec progressively fuses cross-scale NDT features, producing holistic scene tokens consumable by LLM endpoints. Beyond tokenization, MSDec is repurposed as a general interface for human-interactive prompting (points, boxes, masks) and segmentation-mask decoding, unifying diverse 3D scene understanding tasks within a single architecture. With this compact and unified design, NDTokenizer3D offers a fine-grained, general-purpose 3D VLM, achieving remarkable improvements in 3D Referring Segmentation, 3D Visual Question Answering, and 3D Dense Captioning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 64 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

50 of 64 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models22 Jul
  2. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
  3. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences2 Dec
  4. 20243D Vision and Language Pretraining with Large-Scale Synthetic Data8 Jul
  5. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model27 May
  6. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors18 Mar
  7. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding6 Jan
  8. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding21 Dec
  9. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  10. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding18 May
  11. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes12 Apr

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  9. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  10. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  12. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
27 more
  1. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  2. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  3. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  4. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  6. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  7. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  8. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  9. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  10. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  11. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  12. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  13. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  14. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  15. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  16. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  17. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  18. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  19. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  20. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  21. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  22. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 32
  23. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  24. 2025ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoningcited by 25
  25. 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
  26. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  27. 2025LLM-RG: Referential Grounding in Outdoor Scenarios using Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
  2. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
  3. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  4. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs
  5. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  6. 2026Geometry-Guided 3D Visual Token Pruning for Video-Language Models
  7. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
  8. 2025Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization