mlsanityData to 7 Oct 2026

A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

arXiv 2601.22830 ↗First posted 30 Jan 2026cs.CV cs.ROcited by 1

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against two specialized detectors: the anchor-based YOLOv5 and the transformer-based RT-DETRv4. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3) surpass the YOLOv5 in recall by over 25% and closely match RT-DETRv4 under natural visual degradation, while specialized detectors retain an advantage in geometric precision for handcrafted perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 19 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

9 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models11 May

Other references · 8

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  6. 2024YOLO Evolution: A Comprehensive Benchmark and Architectural Review of YOLOv12, YOLO11, and Their Previous Versionscited by 18
  7. 2025RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Modelscited by 13
  8. 2022PeSOTIF: a Challenging Visual Dataset for Perception SOTIF Problems in Long-tail Traffic Scenarioscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026VeriFuse: Bounded Vision-Language Arbitration and Reason-Guided Refinement for Cooperative 3D Perceptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing
  2. 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
  3. 2024Effectiveness Assessment of Recent Large Vision-Language Models
  4. 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends
  5. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
  6. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning
  7. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
  8. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations