# ROAD: The ROad event Awareness Dataset for Autonomous Driving

arXiv: https://arxiv.org/abs/2102.11585 · HTML: https://mlsanity.com/paper/2102.11585/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 121 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 30 of 56)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [The ApolloScape Open Dataset for Autonomous Driving and its Application](https://mlsanity.com/paper/1803.06184/) (2018)
- [CityPersons: A Diverse Dataset for Pedestrian Detection](https://mlsanity.com/paper/1702.05693/) (2017)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)
- [TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions](https://mlsanity.com/paper/1812.04767/) (2018)
- [The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes](https://mlsanity.com/paper/1903.01568/) (2019)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)

## Built on by (most cited first, 30 of 54)

- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping](https://mlsanity.com/paper/2304.10440/) (2023)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review](https://mlsanity.com/paper/2402.10086/) (2024)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [ROAD-R: The Autonomous Driving Dataset with Logical Requirements](https://mlsanity.com/paper/2210.01597/) (2022)
- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future](https://mlsanity.com/paper/2312.03408/) (2023)
- [RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narratives](https://mlsanity.com/paper/2503.21459/) (2025)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [METEOR:A Dense, Heterogeneous, and Unstructured Traffic Dataset With Rare Behaviors](https://mlsanity.com/paper/2109.07648/) (2021)
- [Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes](https://mlsanity.com/paper/2311.17948/) (2023)
- [Explainable Action Prediction through Self-Supervision on Scene Graphs](https://mlsanity.com/paper/2302.03477/) (2023)
- [From Spoken Thoughts to Automated Driving Commentary: Predicting and Explaining Intelligent Vehicles' Actions](https://mlsanity.com/paper/2204.09109/) (2022)
- [Machine Learning with Requirements: a Manifesto](https://mlsanity.com/paper/2304.03674/) (2023)
- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [Argus++: Robust Real-time Activity Detection for Unconstrained Video Streams with Overlapping Cube Proposals](https://mlsanity.com/paper/2201.05290/) (2022)
- [Spatiotemporal Deformable Scene Graphs for Complex Activity Detection](https://mlsanity.com/paper/2104.08194/) (2021)
- [Towards Robust Probabilistic Modeling on SO(3) via Rotation Laplace Distribution](https://mlsanity.com/paper/2305.10465/) (2023)
- [Temporal DINO: A Self-supervised Video Strategy to Enhance Action Prediction](https://mlsanity.com/paper/2308.04589/) (2023)
- [Exploiting T-norms for Deep Learning in Autonomous Driving](https://mlsanity.com/paper/2402.11362/) (2024)
- [Probabilistic Approach for Road-Users Detection](https://mlsanity.com/paper/2112.01360/) (2021)
- [Improving Transfer Learning with a Dual Image and Video Transformer for Multi-label Movie Trailer Genre Classification](https://mlsanity.com/paper/2210.07983/) (2022)
- [Self-Supervised Multi-Frame Neural Scene Flow](https://mlsanity.com/paper/2403.16116/) (2024)
- [Random-Set Graph Neural Networks](https://mlsanity.com/paper/2605.11987/) (2026)
- [Monitoring and Adapting the Physical State of a Camera for Autonomous Vehicles](https://mlsanity.com/paper/2112.05456/) (2021)
- [RALACs: Action Recognition in Autonomous Vehicles using Interaction Encoding and Optical Flow](https://mlsanity.com/paper/2209.14408/) (2022)
- [A Survey on Datasets for Decision-making of Autonomous Vehicle](https://mlsanity.com/paper/2306.16784/) (2023)
- [A Hybrid Graph Network for Complex Activity Detection in Video](https://mlsanity.com/paper/2310.17493/) (2023)

## Abstract (arXiv)

Humans drive in a holistic fashion which entails, in particular, understanding dynamic road events and their evolution. Injecting these capabilities in autonomous vehicles can thus take situational awareness and decision making closer to human-level performance. To this purpose, we introduce the ROad event Awareness Dataset (ROAD) for Autonomous Driving, to our knowledge the first of its kind. ROAD is designed to test an autonomous vehicle's ability to detect road events, defined as triplets composed by an active agent, the action(s) it performs and the corresponding scene locations. ROAD comprises videos originally from the Oxford RobotCar Dataset annotated with bounding boxes showing the location in the image plane of each road event. We benchmark various detection tasks, proposing as a baseline a new incremental algorithm for online road event awareness termed 3D-RetinaNet. We also report the performance on the ROAD tasks of Slowfast and YOLOv5 detectors, as well as that of the winners of the ICCV2021 ROAD challenge, which highlight the challenges faced by situation awareness in autonomous driving. ROAD is designed to allow scholars to investigate exciting tasks such as complex (road) activity detection, future event anticipation and continual learning. The dataset is available at https://github.com/gurkirt/road-dataset; the baseline can be found at https://github.com/gurkirt/3D-RetinaNet.