mlsanityData to 7 Oct 2026

A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

arXiv 2603.28888 ↗First posted 30 Mar 2026cs.ROcited by 0

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Semantic anomalies-context-dependent hazards that pixel-level detectors cannot reason about-pose a critical safety risk in autonomous driving. We propose a semantic observer layer: a quantized vision-language model (VLM) running at 1--2,Hz alongside the primary AV control loop, monitoring for semantic edge cases, and triggering fail-safe handoffs when detected. Using Nvidia Cosmos-Reason1-7B with NVFP4 quantization and FlashAttention2, we achieve ~500 ms inference a ~50x speedup over the unoptimized FP16 baseline (no quantization, standard PyTorch attention) on the same hardware--satisfying the observer timing budget. We benchmark accuracy, latency, and quantization behavior in static and video conditions, identify NF4 recall collapse (10.6%) as a hard deployment constraint, and a hazard analysis mapping performance metrics to safety goals. The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 14 of its 22 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

14 of 22 references.

References · 14

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  5. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  6. 2023Semantic Anomaly Detection with Large Language Modelscited by 53
  7. 2022RDD2022: A multi-national image dataset for automatic Road Damage Detectioncited by 32
  8. 2024Reasoning Multi-Agent Behavioral Topology for Interactive Autonomous Drivingcited by 27
  9. 2024GND: Global Navigation Dataset with Multi-Modal Perception and Multi-Category Traversability in Outdoor Campus Environmentscited by 10
  10. 2024Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metricscited by 8
  11. 2024EnvoDat: A Large-Scale Multisensory Dataset for Robotic Spatial Awareness and Semantic Reasoning in Heterogeneous Environmentscited by 7
  12. 2024Safety Verification for Evasive Collision Avoidance in Autonomous Vehicles with Enhanced Resolutionscited by 3
2 more
  1. 2025CoT-VLM4Tar: Chain-of-Thought Guided Vision-Language Models for Traffic Anomaly Resolutioncited by 3
  2. 2026CausalNav: A Long-term Embodied Navigation System for Autonomous Mobile Robots in Dynamic Outdoor Scenarioscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning
  2. 2026VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
  3. 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
  4. 2025INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluation
  5. 2025LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments
  6. 2026A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions
  7. 2026Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving
  8. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives