# BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

arXiv: https://arxiv.org/abs/2604.07263 · HTML: https://mlsanity.com/paper/2604.07263/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving.
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Placed in

- End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/)

## Builds on (most cited first, 10 of 10)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Driver Anomaly Detection: A Dataset and Contrastive Learning Approach](https://mlsanity.com/paper/2009.14660/) (2020)
- [AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception](https://mlsanity.com/paper/2307.13933/) (2023)
- [DeepTake: Prediction of Driver Takeover Behavior using Multimodal Data](https://mlsanity.com/paper/2012.15441/) (2020)
- [The Multimodal Driver Monitoring Database: A Naturalistic Corpus to Study Driver Attention](https://mlsanity.com/paper/2101.04639/) (2020)
- [ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement](https://mlsanity.com/paper/2603.06986/) (2026)

## Built on by (most cited first, 2 of 2)

- [DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting](https://mlsanity.com/paper/2609.08117/) (2026)
- [TriDrive: Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring](https://mlsanity.com/paper/2609.33000/) (2026)

## Abstract (arXiv)

Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to remain continuously attentive and ready to intervene. This design demands substantial situational judgment and imposes significant cognitive load, leading to steep learning curves, suboptimal user experience, and safety risks from both over-reliance and delayed takeover. Predicting when drivers hand over control to DA and when they take it back is therefore critical for designing proactive, context-aware HMI, yet existing datasets rarely capture the multimodal context, including road scene, driver state, vehicle dynamics, and route environment. To fill this gap, we introduce BATON, a large-scale naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving. The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context, forming a closed-loop multimodal record around each control transition. We define three benchmark tasks: driving action understanding, handover prediction, and takeover prediction, and evaluate baselines spanning sequence models, classical classifiers, and zero-shot VLMs. Results show that visual input alone is insufficient for reliable transition prediction: front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene. Incorporating CAN and route-context signals substantially improves performance over video-only settings, indicating strong complementarity across modalities. We further find takeover events develop more gradually and benefit from longer prediction horizons, whereas handover events depend more on immediate contextual cues, revealing an asymmetry with direct implications for HMI design in assisted driving systems.