# OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

arXiv: https://arxiv.org/abs/2608.17110 · HTML: https://mlsanity.com/paper/2608.17110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 32 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [Open-Vocabulary Point-Cloud Object Detection without 3D Annotation](https://mlsanity.com/paper/2304.00788/) (2023)
- [Better Call SAL: Towards Learning to Segment Anything in Lidar](https://mlsanity.com/paper/2403.13129/) (2024)
- [Towards Long-Tailed 3D Detection](https://mlsanity.com/paper/2211.08691/) (2022)
- [Detect Anything 3D in the Wild](https://mlsanity.com/paper/2504.07958/) (2025)
- [Revisiting Few-Shot Object Detection with Vision-Language Models](https://mlsanity.com/paper/2312.14494/) (2023)
- [Open Vocabulary Monocular 3D Object Detection](https://mlsanity.com/paper/2411.16833/) (2024)
- [Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models](https://mlsanity.com/paper/2505.20612/) (2025)
- [ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images](https://mlsanity.com/paper/2410.24001/) (2024)
- [WildDet3D: Scaling Promptable 3D Detection in the Wild](https://mlsanity.com/paper/2604.08626/) (2026)
- [RefAV: Towards Planning-Centric Scenario Mining](https://mlsanity.com/paper/2505.20981/) (2025)
- [Long-Tailed 3D Detection via Multi-Modal Fusion](https://mlsanity.com/paper/2312.10986/) (2023)
- [Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection](https://mlsanity.com/paper/2406.10115/) (2024)
- [Towards Learning to Complete Anything in Lidar](https://mlsanity.com/paper/2504.12264/) (2025)
- [Modeling Variants of Prompts for Vision-Language Models](https://mlsanity.com/paper/2503.08229/) (2025)
- [User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments](https://mlsanity.com/paper/2601.23281/) (2026)

## Abstract (arXiv)

Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric. To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer. We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D's performance collapses from 18.6 to 5.4 AP when prompted with "a detailed high-resolution photo of a car" rather than "car"); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D's AP by 1.9 × on ScanNet). Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector's predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.