# A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

arXiv: https://arxiv.org/abs/2603.28888 · HTML: https://mlsanity.com/paper/2603.28888/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 14 of 14)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning](https://mlsanity.com/paper/2503.15558/) (2025)
- [Real-Time Anomaly Detection and Reactive Planning with Large Language Models](https://mlsanity.com/paper/2407.08735/) (2024)
- [Semantic Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2305.11307/) (2023)
- [RDD2022: A multi-national image dataset for automatic Road Damage Detection](https://mlsanity.com/paper/2209.08538/) (2022)
- [Reasoning Multi-Agent Behavioral Topology for Interactive Autonomous Driving](https://mlsanity.com/paper/2409.18031/) (2024)
- [GND: Global Navigation Dataset with Multi-Modal Perception and Multi-Category Traversability in Outdoor Campus Environments](https://mlsanity.com/paper/2409.14262/) (2024)
- [Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics](https://mlsanity.com/paper/2401.04942/) (2024)
- [EnvoDat: A Large-Scale Multisensory Dataset for Robotic Spatial Awareness and Semantic Reasoning in Heterogeneous Environments](https://mlsanity.com/paper/2410.22200/) (2024)
- [Safety Verification for Evasive Collision Avoidance in Autonomous Vehicles with Enhanced Resolutions](https://mlsanity.com/paper/2411.02706/) (2024)
- [CoT-VLM4Tar: Chain-of-Thought Guided Vision-Language Models for Traffic Anomaly Resolution](https://mlsanity.com/paper/2503.01632/) (2025)
- [CausalNav: A Long-term Embodied Navigation System for Autonomous Mobile Robots in Dynamic Outdoor Scenarios](https://mlsanity.com/paper/2601.01872/) (2026)

## Abstract (arXiv)

Semantic anomalies-context-dependent hazards that pixel-level detectors cannot reason about-pose a critical safety risk in autonomous driving. We propose a semantic observer layer: a quantized vision-language model (VLM) running at 1--2,Hz alongside the primary AV control loop, monitoring for semantic edge cases, and triggering fail-safe handoffs when detected. Using Nvidia Cosmos-Reason1-7B with NVFP4 quantization and FlashAttention2, we achieve ~500 ms inference a ~50x speedup over the unoptimized FP16 baseline (no quantization, standard PyTorch attention) on the same hardware--satisfying the observer timing budget. We benchmark accuracy, latency, and quantization behavior in static and video conditions, identify NF4 recall collapse (10.6%) as a hard deployment constraint, and a hazard analysis mapping performance metrics to safety goals. The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms.