# NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

arXiv: https://arxiv.org/abs/2608.14767 · HTML: https://mlsanity.com/paper/2608.14767/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail](https://mlsanity.com/paper/2511.00088/) (2025)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2504.04348/) (2025)
- [Talk2Car: Taking Control of Your Self-Driving Car](https://mlsanity.com/paper/1909.10838/) (2019)
- [Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving](https://mlsanity.com/paper/2312.03661/) (2023)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Explainable Object-induced Action Decision for Autonomous Vehicles](https://mlsanity.com/paper/2003.09405/) (2020)
- [CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving](https://mlsanity.com/paper/2408.10845/) (2024)
- [Grounding Human-to-Vehicle Advice for Self-driving Vehicles](https://mlsanity.com/paper/1911.06978/) (2019)
- [LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs](https://mlsanity.com/paper/2312.04372/) (2023)
- [Situational Awareness, Drivers Trust in Automated Driving Systems and Secondary Task Performance](https://mlsanity.com/paper/1903.05251/) (2019)
- [Saliency-Guided Domain Adaptation for Left-Hand Driving in Autonomous Steering](https://mlsanity.com/paper/2511.01223/) (2025)
- [X-Blocks: Linguistic Building Blocks of Natural Language Explanations for Automated Vehicles](https://mlsanity.com/paper/2602.13248/) (2026)

## Abstract (arXiv)

Automated vehicles must explain their decisions in ways that passengers can understand, monitor, and trust. Existing language-annotated driving datasets are mostly observer-written, post-hoc, simulation-based, or generated from sensor inputs, rather than elicited from the driver performing the action. We introduce NARRATE, a multimodal real-world Australian driving dataset comprising 2,050 annotated events from 35 experienced drivers and driving instructors on public roads. Each event is grounded in synchronised visual, localisation, motion, and LiDAR streams and paired with in-vehicle and/or post-drive free-text explanations. NARRATE provides action labels, scenario-context labels spanning six high-level and 32 fine-grained categories, and span-level Situational Awareness (SA) annotations over driver explanations for Perception, Comprehension and Projection. Four benchmark tasks (SA, scenario-context, driver-action classification, and explanation generation) show that this structure is learnable from driver language, while fine-grained context recognition and explanation generation remain challenging. NARRATE paves a path towards more human-centred and domain-aware explanation models for automated driving.