# Auto-Vocabulary 3D Object Detection

arXiv: https://arxiv.org/abs/2512.16077 · HTML: https://mlsanity.com/paper/2512.16077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 60 of its 72 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 30 of 60)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023)
- [Delving into Localization Errors for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.16237/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://mlsanity.com/paper/2006.05682/) (2020)
- [Open-Vocabulary Point-Cloud Object Detection without 3D Annotation](https://mlsanity.com/paper/2304.00788/) (2023)
- [MLCVNet: Multi-Level Context VoteNet for 3D Object Detection](https://mlsanity.com/paper/2004.05679/) (2020)
- [CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data](https://mlsanity.com/paper/2303.12417/) (2023)
- [Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2104.06114/) (2021)
- [CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection](https://mlsanity.com/paper/2310.02960/) (2023)
- [MonoCD: Monocular 3D Object Detection with Complementary Depths](https://mlsanity.com/paper/2404.03181/) (2024)
- [InfoFocus: 3D Object Detection for Autonomous Driving with Dynamic Information Modeling](https://mlsanity.com/paper/2007.08556/) (2020)
- [Vocabulary-free Image Classification](https://mlsanity.com/paper/2306.00917/) (2023)
- [DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2505.04410/) (2025)
- [InstaGen: Enhancing Object Detection by Training on Synthetic Dataset](https://mlsanity.com/paper/2402.05937/) (2024)
- [Taming Self-Training for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2308.06412/) (2023)
- [Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection](https://mlsanity.com/paper/2406.00510/) (2024)
- [Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning](https://mlsanity.com/paper/2311.11646/) (2023)

## Abstract (arXiv)

Open-vocabulary 3D object detection methods are able to localize 3D boxes of classes unseen during training. Despite the name, existing methods rely on user-specified classes both at training and inference. We propose to study Auto-Vocabulary 3D Object Detection (AV3DOD), where the classes are automatically generated for the detected objects without any user input. To this end, we introduce Semantic Score (SS) to evaluate the quality of the generated class names. We then develop a novel framework, AV3DOD, which leverages 2D vision-language models (VLMs) to generate rich semantic candidates through image captioning, pseudo 3D box generation, and feature-space semantics expansion. AV3DOD achieves the state-of-the-art (SOTA) performance on both localization (mAP) and semantic quality (SS) on the ScanNetV2 and SUNRGB-D datasets. Notably, it surpasses the SOTA, CoDA, by 3.48 overall mAP and attains a 24.5% relative improvement in SS on ScanNetV2.