# SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events

arXiv: https://arxiv.org/abs/2103.15538 · HTML: https://mlsanity.com/paper/2103.15538/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 69 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Exploring Models and Data for Image Question Answering](https://mlsanity.com/paper/1505.02074/) (2015)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition](https://mlsanity.com/paper/1904.04289/) (2019)

## Built on by (most cited first, 30 of 84)

- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.08052/) (2025)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding](https://mlsanity.com/paper/2406.09418/) (2024)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model](https://mlsanity.com/paper/2501.12368/) (2025)
- [Number it: Temporal Grounding Videos like Flipping Manga](https://mlsanity.com/paper/2411.10332/) (2024)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions](https://mlsanity.com/paper/2412.09596/) (2024)
- [Abductive Ego-View Accident Video Understanding for Safe Driving Perception](https://mlsanity.com/paper/2403.00436/) (2024)
- [FunQA: Towards Surprising Video Comprehension](https://mlsanity.com/paper/2306.14899/) (2023)
- [Vision-Based Traffic Accident Detection and Anticipation: A Survey](https://mlsanity.com/paper/2308.15985/) (2023)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers](https://mlsanity.com/paper/2407.08394/) (2024)
- [Reading Between the Lanes: Text VideoQA on the Road](https://mlsanity.com/paper/2307.03948/) (2023)
- [CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process](https://mlsanity.com/paper/2401.14535/) (2024)
- [Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer](https://mlsanity.com/paper/2308.08414/) (2023)
- [Uncovering Main Causalities for Long-tailed Information Extraction](https://mlsanity.com/paper/2109.05213/) (2021)
- [AVD2: Accident Video Diffusion for Accident Video Description](https://mlsanity.com/paper/2502.14801/) (2025)
- [VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding](https://mlsanity.com/paper/2507.09815/) (2025)
- [DAM: Dynamic Adapter Merging for Continual Video QA Learning](https://mlsanity.com/paper/2403.08755/) (2024)
- [Video Question Answering with Iterative Video-Text Co-Tokenization](https://mlsanity.com/paper/2208.00934/) (2022)
- [Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark](https://mlsanity.com/paper/2212.09381/) (2022)
- [Explaining Autonomous Driving Actions with Visual Question Answering](https://mlsanity.com/paper/2307.10408/) (2023)

## Abstract (arXiv)

Traffic event cognition and reasoning in videos is an important task that has a wide range of applications in intelligent transportation, assisted driving, and autonomous vehicles. In this paper, we create a novel dataset, SUTD-TrafficQA (Traffic Question Answering), which takes the form of video QA based on the collected 10,080 in-the-wild videos and annotated 62,535 QA pairs, for benchmarking the cognitive capability of causal inference and event understanding models in complex traffic scenarios. Specifically, we propose 6 challenging reasoning tasks corresponding to various traffic scenarios, so as to evaluate the reasoning capability over different kinds of complex yet practical traffic events. Moreover, we propose Eclipse, a novel Efficient glimpse network via dynamic inference, in order to achieve computation-efficient and reliable video reasoning. The experiments show that our method achieves superior performance while reducing the computation cost significantly. The project page: https://github.com/SUTDCV/SUTD-TrafficQA.