# Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation

arXiv: https://arxiv.org/abs/2107.12932 · HTML: https://mlsanity.com/paper/2107.12932/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 86 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture](https://mlsanity.com/paper/1509.05016/) (2015)
- [Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis](https://mlsanity.com/paper/1802.02690/) (2018)
- [Dynamics of Driver's Gaze: Explorations in Behavior Modeling & Maneuver Prediction](https://mlsanity.com/paper/1802.00066/) (2018)
- [Looking at the Driver/Rider in Autonomous Vehicles to Predict Take-Over Readiness](https://mlsanity.com/paper/1811.06047/) (2018)
- [DeepTake: Prediction of Driver Takeover Behavior using Multimodal Data](https://mlsanity.com/paper/2012.15441/) (2020)
- [Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data](https://mlsanity.com/paper/2104.11489/) (2021)
- [Owl and Lizard: Patterns of Head Pose and Eye Pose in Driver Gaze Classification](https://mlsanity.com/paper/1508.04028/) (2015)
- [Looking at Hands in Autonomous Vehicles: A ConvNet Approach using Part Affinity Fields](https://mlsanity.com/paper/1804.01176/) (2018)
- [HandyNet: A One-stop Solution to Detect, Segment, Localize & Analyze Driver Hands](https://mlsanity.com/paper/1804.07834/) (2018)
- [Forced Spatial Attention for Driver Foot Activity Classification](https://mlsanity.com/paper/1907.11824/) (2019)

## Built on by (most cited first, 10 of 10)

- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation](https://mlsanity.com/paper/2412.05893/) (2024)
- [Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction](https://mlsanity.com/paper/2301.05805/) (2023)
- [Stochasticity in Motion: An Information-Theoretic Approach to Trajectory Prediction](https://mlsanity.com/paper/2410.01628/) (2024)
- [A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects](https://mlsanity.com/paper/2506.19769/) (2025)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)
- [Driver Activity Classification Using Generalizable Representations from Vision-Language Models](https://mlsanity.com/paper/2404.14906/) (2024)
- [Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras](https://mlsanity.com/paper/2403.00196/) (2024)
- [Robust Detection, Association, and Localization of Vehicle Lights: A Context-Based Cascaded CNN Approach and Evaluations](https://mlsanity.com/paper/2307.14571/) (2023)
- [Vision-based Analysis of Driver Activity and Driving Performance Under the Influence of Alcohol](https://mlsanity.com/paper/2309.08021/) (2023)

## Abstract (arXiv)

Understanding occupant-vehicle interactions by modeling control transitions is important to ensure safe approaches to passenger vehicle automation. Models which contain contextual, semantically meaningful representations of driver states can be used to determine the appropriate timing and conditions for transfer of control between driver and vehicle. However, such models rely on real-world control take-over data from drivers engaged in distracting activities, which is costly to collect. Here, we introduce a scheme for data augmentation for such a dataset. Using the augmented dataset, we develop and train take-over time (TOT) models that operate sequentially on mid and high-level features produced by computer vision algorithms operating on different driver-facing camera views, showing models trained on the augmented dataset to outperform the initial dataset. The demonstrated model features encode different aspects of the driver state, pertaining to the face, hands, foot and upper body of the driver. We perform ablative experiments on feature combinations as well as model architectures, showing that a TOT model supported by augmented data can be used to produce continuous estimates of take-over times without delay, suitable for complex real-world scenarios.