TAU-R1: Visual Language Model for Traffic Anomaly Understanding
On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection
Abstract (arXiv)
Traffic Anomaly Understanding (TAU) is important for traffic safety in Intelligent Transportation Systems. Recent vision-language models (VLMs) have shown strong capabilities in video understanding. However, progress on TAU remains limited due to the lack of benchmarks and task-specific methodologies. To address this limitation, we introduce Roundabout-TAU, a dataset constructed from real-world roundabout videos collected in collaboration with the City of Carmel, Indiana. The dataset contains 342 clips and is annotated with more than 2,000 question-answer pairs covering multiple aspects of traffic anomaly understanding. Building on this benchmark, we propose TAU-R1, a two-layer vision-language framework for TAU. The first layer is a lightweight anomaly classifier that performs coarse anomaly categorisation, while the second layer is a larger anomaly reasoner that generates detailed event summaries. To improve task-specific reasoning, we introduce a two-stage training strategy consisting of decomposed-QA-enhanced supervised fine-tuning followed by TAU-GRPO, a GRPO-based post-training method with TAU-specific reward functions. Experimental results show that TAU-R1 achieves strong performance on both anomaly classification and reasoning tasks while maintaining deployment efficiency. The dataset and code are available at: https://github.com/siri-rouser/TAU-R1
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 59 references that are in the map. First posted on arXiv 19 Mar 2026.
Builds on
41 of 59 references.
References · 41
- 2023GPT-4 Technical Reportcited by 18,132
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Visual Instruction Tuningcited by 7,792
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
- 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
- 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
29 more
- 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
- 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
- 2020Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learningcited by 91
- 2021UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detectioncited by 91
- 2024Harnessing Large Language Models for Training-free Video Anomaly Detectioncited by 85
- 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
- 2019Unsupervised Traffic Accident Detection in First-Person Videoscited by 69
- 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 66
- 2024Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLMcited by 63
- 2024Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Modelscited by 55
- 2024Advancing Video Anomaly Detection: A Concise Review and a New Datasetcited by 48
- 2024Hawk: Learning to Understand Open-World Video Anomaliescited by 47
- 2024Video Anomaly Detection and Explanation via Large Language Modelscited by 45
- 2024Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomalycited by 44
- 2023Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challengescited by 42
- 2025EventVAD: Training-Free Event-Aware Video Anomaly Detectioncited by 32
- 2024VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Modelscited by 24
- 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 23
- 2022TAD: A Large-Scale Benchmark for Traffic Accidents Detection from Video Surveillancecited by 20
- 2025AnyAnomaly: Zero-Shot Customizable Video Anomaly Detection with LVLMcited by 15
- 2025VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Treecited by 15
- 2025MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detectioncited by 10
- 2025LAD-Reasoner: Tiny Multimodal Models are Good Reasoners for Logical Anomaly Detectioncited by 9
- 2025A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysiscited by 9
- 2025Real-time Traffic Accident Anticipation with Feature Reusecited by 8
- 2023MM-AU:Towards Multimodal Understanding of Advertisement Videoscited by 7
- 2025Aligning Effective Tokens with Video Anomaly in Large Language Modelscited by 7
- 2025SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Modelcited by 6
- 2025RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation Systemcited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning
- 2024GPT-4V as Traffic Assistant: An In-depth Look at Vision Language Model on Complex Traffic Events
- 2026UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations
- 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis
- 2025TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
- 2025VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning
- 2025CoT-VLM4Tar: Chain-of-Thought Guided Vision-Language Models for Traffic Anomaly Resolution
- 2025Structured Prompting and Multi-Agent Knowledge Distillation for Traffic Video Interpretation and Risk Inference