mlsanityData to 14 Oct 2026

On the Potential of Open-Vocabulary Models for Object Detection in Unusual Street Scenes

arXiv 2408.11221 ↗First posted 20 Aug 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Out-of-distribution (OOD) object detection is a critical task focused on detecting objects that originate from a data distribution different from that of the training data. In this study, we investigate to what extent state-of-the-art open-vocabulary object detectors can detect unusual objects in street scenes, which are considered as OOD or rare scenarios with respect to common street scene datasets. Specifically, we evaluate their performance on the OoDIS Benchmark, which extends RoadAnomaly21 and RoadObstacle21 from SegmentMeIfYouCan, as well as LostAndFound, which was recently extended to object level annotations. The objective of our study is to uncover short-comings of contemporary object detectors in challenging real-world, and particularly in open-world scenarios. Our experiments reveal that open vocabulary models are promising for OOD object detection scenarios, however far from perfect. Substantial improvements are required before they can be reliably deployed in real-world applications. We benchmark four state-of-the-art open-vocabulary object detection models on three different datasets. Noteworthily, Grounding DINO achieves the best results on RoadObstacle21 and LostAndFound in our study with an AP of 48.3% and 25.4% respectively. YOLO-World excels on RoadAnomaly21 with an AP of 21.2%.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 47 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

43 of 47 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  11. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
31 more
  1. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  3. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  4. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  5. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  7. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  8. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  9. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  10. 2021Trustworthy AI: From Principles to Practicescited by 207
  11. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 171
  12. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 152
  13. 2021SegmentMeIfYouCan: A Benchmark for Anomaly Segmentationcited by 150
  14. 2018Evaluating Bayesian Deep Learning Methods for Semantic Segmentationcited by 143
  15. 2019Detecting the Unexpected via Image Resynthesiscited by 137
  16. 2020Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentationcited by 133
  17. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  18. 2016Lost and Found: Detecting Small Road Hazards for Self-Driving Vehiclescited by 117
  19. 2021Pixel-wise Anomaly Detection in Complex Driving Scenescited by 109
  20. 2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 59
  21. 2023Semantic Anomaly Detection with Large Language Modelscited by 53
  22. 2023Unmasking Anomalies in Road-Scene Segmentationcited by 45
  23. 2024Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomalycited by 45
  24. 2022SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detectioncited by 33
  25. 2024Customizing Visual-Language Foundation Models for Multi-modal Anomaly Detection and Reasoningcited by 20
  26. 2022UQGAN: A Unified Model for Uncertainty Quantification of Deep Classifiers trained via Conditional GANscited by 16
  27. 2023UGainS: Uncertainty Guided Anomaly Instance Segmentationcited by 15
  28. 2024Anomaly Detection of Tabular Data Using LLMscited by 12
  29. 2022Segmenting Known Objects and Unseen Unknowns without Prior Knowledgecited by 9
  30. 2024OoDIS: Anomaly Instance Segmentation and Detection Benchmarkcited by 8
  31. 2024CLIP-driven Outliers Synthesis for few-shot OOD detectioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Drivingcited by 2
  2. 2025Can We Challenge Open-Vocabulary Object Detectors with Generated Content in Street Scenes?cited by 1
  3. 2025CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoningcited by 0
  4. 2026Out-of-Distribution Object Detection in Street Scenes via Synthetic Outlier Exposure and Transfer Learningcited by 0
  5. 2026DSAA: Dual-Stage Attribute Activation for Fine-grained Open Vocabulary Detectioncited by 0