# Task-Aligned Stability Analysis of Vision-Language Models for Autonomous Driving Hazard Detection

arXiv: https://arxiv.org/abs/2606.11889 · HTML: https://mlsanity.com/paper/2606.11889/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 5 of its 13 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 5 of 5)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Using Vision Language Models for Safety Hazard Identification in Construction](https://mlsanity.com/paper/2504.09083/) (2025)
- [KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System](https://mlsanity.com/paper/2512.20299/) (2025)
- [Edge-Enabled Collaborative Object Detection for Real-Time Multi-Vehicle Perception](https://mlsanity.com/paper/2506.06474/) (2025)

## Abstract (arXiv)

Vision-language models (VLMs) are increasingly used for scene understanding in autonomous driving, but robustness analysis often relies on task-agnostic embedding stability alone. We study whether corruption-induced embedding drift predicts changes in a task-aligned hazard score derived from CLIP image-text similarities. Using controlled corruptions on BDD100K road scenes, we compare embedding drift against margin drift, defined as the change in hazard score under perturbation. The relationship is highly corruption-dependent: some families exhibit strong coupling between representation drift and decision drift, while others induce hazardous decision instability despite relatively modest embedding change. Furthermore, corruption families differ in failure direction: most suppress hazard detections via false negatives, while occlusion instead triggers false alarms, suggesting that benchmark design should account for asymmetric failure modes, not just overall instability rates. These results suggest that robustness benchmarks should include task-aligned stability measures in addition to embedding-level perturbation statistics.