mlsanityData to 14 Oct 2026

OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

arXiv 2601.09575 ↗First posted 14 Jan 2026cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 80 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

72 of 80 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025CCL-LGS: Contrastive Codebook Learning for 3D Language Gaussian Splatting26 May
  2. 20243D Vision-Language Gaussian Splatting10 Oct

Other references · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023Segment Anythingcited by 8,234
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  12. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
38 more
  1. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  2. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  3. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  4. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  5. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
  6. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  7. 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 667
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2023LERF: Language Embedded Radiance Fieldscited by 543
  10. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  11. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  12. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  13. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 364
  14. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
  15. 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 286
  16. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  17. 2024A Survey on 3D Gaussian Splattingcited by 224
  18. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  19. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  20. 2021GeoNeRF: Generalizing NeRF with Geometry Priorscited by 196
  21. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  22. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  23. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  24. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  25. 2022Generalizable Patch-Based Neural Renderingcited by 122
  26. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  27. 2019Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidthcited by 89
  28. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  29. 2023OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learningcited by 75
  30. 2022Removing Objects From Neural Radiance Fieldscited by 66
  31. 2023NeRFiller: Completing Scenes via Generative 3D Inpaintingcited by 62
  32. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Priorcited by 56
  33. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  34. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  35. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 51
  36. 2024Gaga: Group Any Gaussians via 3D-aware Memory Bankcited by 41
  37. 2023Reference-guided Controllable Inpainting of Neural Radiance Fieldscited by 40
  38. 2024GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removalcited by 39

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026GeoNVS: Geometry Grounded Video Diffusion for Novel View Synthesiscited by 3
  2. 2026LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understandingcited by 1
  3. 2026MAG-3D: Multi-Agent Grounded Reasoning for 3D Understandingcited by 0
  4. 2026FoundObj: Self-supervised Foundation Models as Rewards for Label-free 3D Object Segmentationcited by 0
  5. 2026SafeVantage: Vantage-Aware Memory for Reliable Embodied Decisionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenVox: Real-time Instance-level Open-vocabulary Probabilistic Voxel Representation
  2. 2025NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embedding
  3. 2024Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding
  4. 2026Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors
  5. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding
  6. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding
  7. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  8. 2025VoxRep: Enhancing 3D Spatial Understanding in 2D Vision-Language Models via Voxel Representation