# TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes

arXiv: https://arxiv.org/abs/2502.02449 · HTML: https://mlsanity.com/paper/2502.02449/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2025.

## Built on by (most cited first, 23 of 23)

- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding](https://mlsanity.com/paper/2507.09815/) (2025)
- [STER-VLM: Spatio-Temporal With Enhanced Reference Vision-Language Models](https://mlsanity.com/paper/2508.13470/) (2025)
- [V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving](https://mlsanity.com/paper/2505.00156/) (2025)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving](https://mlsanity.com/paper/2509.25944/) (2025)
- [RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System](https://mlsanity.com/paper/2511.18286/) (2025)
- [CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning](https://mlsanity.com/paper/2604.08457/) (2026)
- [Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception](https://mlsanity.com/paper/2510.08352/) (2025)
- [Understanding Real-World Traffic Safety through RoadSafe365 Benchmark](https://mlsanity.com/paper/2602.07212/) (2026)
- [Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Dataset](https://mlsanity.com/paper/2604.22260/) (2026)
- [InterAct-Video: Reasoning-Rich Video QA for Urban Traffic](https://mlsanity.com/paper/2507.14743/) (2025)
- [UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics](https://mlsanity.com/paper/2602.21137/) (2026)
- [ForeSea: AI Forensic Search with Multi-modal Queries for Video Surveillance](https://mlsanity.com/paper/2603.22872/) (2026)
- [Descriptor: Distance-Annotated Traffic Perception Question Answering (DTPQA)](https://mlsanity.com/paper/2511.13397/) (2025)
- [Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimation](https://mlsanity.com/paper/2512.02920/) (2025)
- [SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding](https://mlsanity.com/paper/2604.03697/) (2026)
- [What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs](https://mlsanity.com/paper/2606.01624/) (2026)
- [KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation](https://mlsanity.com/paper/2606.16256/) (2026)
- [CAViAR: A Causal Video Dataset for Fine-Grained Accident Reasoning in Real-World Scenarios](https://mlsanity.com/paper/2608.19380/) (2026)
- [CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning](https://mlsanity.com/paper/2609.35823/) (2026)
- [Vision-Language-Action Autonomous Driving Agent with Language-based Memory](https://mlsanity.com/paper/2609.38641/) (2026)

## Abstract (arXiv)

We present TUMTraffic-VideoQA, a novel dataset and benchmark designed for spatio-temporal video understanding in complex roadside traffic scenarios. The dataset comprises 1,000 videos, featuring 85,000 multiple-choice QA pairs, 2,300 object captioning, and 5,700 object grounding annotations, encompassing diverse real-world conditions such as adverse weather and traffic anomalies. By incorporating tuple-based spatio-temporal object expressions, TUMTraffic-VideoQA unifies three essential tasks-multiple-choice video question answering, referred object captioning, and spatio-temporal object grounding-within a cohesive evaluation framework. We further introduce the TUMTraffic-Qwen baseline model, enhanced with visual token sampling strategies, providing valuable insights into the challenges of fine-grained spatio-temporal reasoning. Extensive experiments demonstrate the dataset's complexity, highlight the limitations of existing models, and position TUMTraffic-VideoQA as a robust foundation for advancing research in intelligent transportation systems. The dataset and benchmark are publicly available to facilitate further exploration.