# Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data

arXiv: https://arxiv.org/abs/2104.11489 · HTML: https://mlsanity.com/paper/2104.11489/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis](https://mlsanity.com/paper/1802.02690/) (2018)
- [Looking at the Driver/Rider in Autonomous Vehicles to Predict Take-Over Readiness](https://mlsanity.com/paper/1811.06047/) (2018)
- [Behavioral Research and Practical Models of Drivers' Attention](https://mlsanity.com/paper/2104.05677/) (2021)
- [Looking at Hands in Autonomous Vehicles: A ConvNet Approach using Part Affinity Fields](https://mlsanity.com/paper/1804.01176/) (2018)
- [Forced Spatial Attention for Driver Foot Activity Classification](https://mlsanity.com/paper/1907.11824/) (2019)
- [Gaze Preserving CycleGANs for Eyeglass Removal & Persistent Gaze Estimation](https://mlsanity.com/paper/2002.02077/) (2020)

## Built on by (most cited first, 13 of 13)

- [doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation](https://mlsanity.com/paper/2412.05893/) (2024)
- [On Salience-Sensitive Sign Classification in Autonomous Vehicle Path Planning: Experimental Explorations with a Novel Dataset](https://mlsanity.com/paper/2112.00942/) (2021)
- [Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction](https://mlsanity.com/paper/2301.05805/) (2023)
- [Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation](https://mlsanity.com/paper/2107.12932/) (2021)
- [(Safe) SMART Hands: Hand Activity Analysis and Distraction Alerts Using a Multi-Camera Framework](https://mlsanity.com/paper/2301.05838/) (2023)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)
- [Driver Activity Classification Using Generalizable Representations from Vision-Language Models](https://mlsanity.com/paper/2404.14906/) (2024)
- [Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras](https://mlsanity.com/paper/2403.00196/) (2024)
- [Robust Detection, Association, and Localization of Vehicle Lights: A Context-Based Cascaded CNN Approach and Evaluations](https://mlsanity.com/paper/2307.14571/) (2023)
- [Vision-based Analysis of Driver Activity and Driving Performance Under the Influence of Alcohol](https://mlsanity.com/paper/2309.08021/) (2023)
- [Creativity and Visual Communication from Machine to Musician: Sharing a Score through a Robotic Camera](https://mlsanity.com/paper/2409.05773/) (2024)
- [Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making](https://mlsanity.com/paper/2602.07668/) (2026)
- [Vision-Language Work Zone Intelligence for Safety-Critical Speed Regulation of Mixed-Autonomy Vehicles in Dynamic Environments](https://mlsanity.com/paper/2606.08860/) (2026)

## Abstract (arXiv)

With increasing automation in passenger vehicles, the study of safe and smooth occupant-vehicle interaction and control transitions is key. In this study, we focus on the development of contextual, semantically meaningful representations of the driver state, which can then be used to determine the appropriate timing and conditions for transfer of control between driver and vehicle. To this end, we conduct a large-scale real-world controlled data study where participants are instructed to take-over control from an autonomous agent under different driving conditions while engaged in a variety of distracting activities. These take-over events are captured using multiple driver-facing cameras, which when labelled result in a dataset of control transitions and their corresponding take-over times (TOTs). We then develop and train TOT models that operate sequentially on mid to high-level features produced by computer vision algorithms operating on different driver-facing camera views. The proposed TOT model produces continuous predictions of take-over times without delay, and shows promising qualitative and quantitative results in complex real-world scenarios.