# Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

arXiv: https://arxiv.org/abs/2610.00855 · HTML: https://mlsanity.com/paper/2610.00855/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Towards Label-free Scene Understanding by Vision Foundation Models](https://mlsanity.com/paper/2306.03899/) (2023)
- [Using a Waffle Iron for Automotive Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2301.10100/) (2023)
- [Better Call SAL: Towards Learning to Segment Anything in Lidar](https://mlsanity.com/paper/2403.13129/) (2024)
- [Three Pillars improving Vision Foundation Model Distillation for Lidar](https://mlsanity.com/paper/2310.17504/) (2023)
- [Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation](https://mlsanity.com/paper/2407.13362/) (2024)
- [AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction](https://mlsanity.com/paper/2412.18255/) (2024)
- [3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving](https://mlsanity.com/paper/2405.15286/) (2024)
- [3D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentation](https://mlsanity.com/paper/2406.09126/) (2024)
- [LOSC: LiDAR Open-voc Segmentation Consolidator](https://mlsanity.com/paper/2507.07605/) (2025)
- [UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervision](https://mlsanity.com/paper/2412.18131/) (2024)
- [SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2604.20395/) (2026)

## Abstract (arXiv)

Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it into a 3D network. These methods rely almost exclusively on voxel-based sparse convolutions, and point transformers have so far been limited to indoor environments, where 3D data is dense and bounded. We present Lang3DSeg, which establishes a point transformer as the backbone for annotation-free open-vocabulary segmentation of outdoor 3D LiDAR, and is trained from scratch without geometric pre-training. This training paradigm necessitates addressing the inherent noise in 2D-to-3D label projections; specifically, naive projection often suffers from depth ambiguity, where points behind an object are erroneously assigned its semantic label. We therefore composite masks using an explicit class-priority rule and truncate each projected instance at the first gap in its depth distribution, correcting the projection error directly rather than averaging it over registered sequences. Lang3DSeg achieves 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI, the highest among published annotation-free methods on both benchmarks. Every 3D semantic segmentation is on a single LiDAR sweep, and inference operates in real-time without running vision-language models.