# Traffic-Domain Video Question Answering with Automatic Captioning

arXiv: https://arxiv.org/abs/2307.09636 · HTML: https://mlsanity.com/paper/2307.09636/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 37 of its 56 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling](https://mlsanity.com/paper/2111.12681/) (2021)
- [UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://mlsanity.com/paper/2201.04676/) (2022)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [ASER: A Large-scale Eventuality Knowledge Graph](https://mlsanity.com/paper/1905.00270/) (2019)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [HoloClean: Holistic Data Repairs with Probabilistic Inference](https://mlsanity.com/paper/1702.00820/) (2017)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Label-Free Supervision of Neural Networks with Physics and Domain Knowledge](https://mlsanity.com/paper/1609.05566/) (2016)
- [Curriculum Learning of Multiple Tasks](https://mlsanity.com/paper/1412.1353/) (2014)
- [Neural Ranking Models with Weak Supervision](https://mlsanity.com/paper/1704.08803/) (2017)
- [QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking](https://mlsanity.com/paper/2210.06984/) (2022)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training](https://mlsanity.com/paper/2109.05003/) (2021)
- [Temporal Common Sense Acquisition with Minimal Supervision](https://mlsanity.com/paper/2005.04304/) (2020)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering](https://mlsanity.com/paper/1910.14087/) (2019)
- [Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision](https://mlsanity.com/paper/2012.08012/) (2020)
- [Advanced Semantics for Commonsense Knowledge Extraction](https://mlsanity.com/paper/2011.00905/) (2020)
- [Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark](https://mlsanity.com/paper/2212.09381/) (2022)

## Built on by (most cited first, 10 of 10)

- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding](https://mlsanity.com/paper/2507.09815/) (2025)
- [TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning](https://mlsanity.com/paper/2404.09275/) (2024)
- [RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narratives](https://mlsanity.com/paper/2503.21459/) (2025)
- [Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding](https://mlsanity.com/paper/2407.05910/) (2024)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Dataset](https://mlsanity.com/paper/2604.22260/) (2026)
- [InterAct-Video: Reasoning-Rich Video QA for Urban Traffic](https://mlsanity.com/paper/2507.14743/) (2025)

## Abstract (arXiv)

Video Question Answering (VidQA) exhibits remarkable potential in facilitating advanced machine reasoning capabilities within the domains of Intelligent Traffic Monitoring and Intelligent Transportation Systems. Nevertheless, the integration of urban traffic scene knowledge into VidQA systems has received limited attention in previous research endeavors. In this work, we present a novel approach termed Traffic-domain Video Question Answering with Automatic Captioning (TRIVIA), which serves as a weak-supervision technique for infusing traffic-domain knowledge into large video-language models. Empirical findings obtained from the SUTD-TrafficQA task highlight the substantial enhancements achieved by TRIVIA, elevating the accuracy of representative video-language models by a remarkable 6.5 points (19.88%) compared to baseline settings. This pioneering methodology holds great promise for driving advancements in the field, inspiring researchers and practitioners alike to unlock the full potential of emerging video-language models in traffic-related applications.