# Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction

arXiv: https://arxiv.org/abs/2301.05805 · HTML: https://mlsanity.com/paper/2301.05805/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 14 Jan 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Trajectory Prediction in Autonomous Driving with a Lane Heading Auxiliary Loss](https://mlsanity.com/paper/2011.06679/) (2020)
- [Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis](https://mlsanity.com/paper/1802.02690/) (2018)
- [Looking at the Driver/Rider in Autonomous Vehicles to Predict Take-Over Readiness](https://mlsanity.com/paper/1811.06047/) (2018)
- [Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data](https://mlsanity.com/paper/2104.11489/) (2021)
- [Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation](https://mlsanity.com/paper/2107.12932/) (2021)
- [Looking at Hands in Autonomous Vehicles: A ConvNet Approach using Part Affinity Fields](https://mlsanity.com/paper/1804.01176/) (2018)
- [HandyNet: A One-stop Solution to Detect, Segment, Localize & Analyze Driver Hands](https://mlsanity.com/paper/1804.07834/) (2018)
- [Forced Spatial Attention for Driver Foot Activity Classification](https://mlsanity.com/paper/1907.11824/) (2019)
- [From Pedestrian Detection to Crosswalk Estimation: An EM Algorithm and Analysis on Diverse Datasets](https://mlsanity.com/paper/2205.12579/) (2022)

## Built on by (most cited first, 12 of 12)

- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation](https://mlsanity.com/paper/2412.05893/) (2024)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)
- [Driver Activity Classification Using Generalizable Representations from Vision-Language Models](https://mlsanity.com/paper/2404.14906/) (2024)
- [Evaluating Driver Readiness in Conditionally Automated Vehicles from Eye-Tracking Data and Head Pose](https://mlsanity.com/paper/2401.11284/) (2024)
- [Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras](https://mlsanity.com/paper/2403.00196/) (2024)
- [Towards Vision Zero: The TUM Traffic Accid3nD Dataset](https://mlsanity.com/paper/2503.12095/) (2025)
- [Enhancing Highway Safety: Accident Detection on the A9 Test Stretch Using Roadside Sensors](https://mlsanity.com/paper/2502.00402/) (2025)
- [Safety-Critical Learning for Long-Tail Events: The TUM Traffic Accident Dataset](https://mlsanity.com/paper/2508.14567/) (2025)
- [Vision-based Analysis of Driver Activity and Driving Performance Under the Influence of Alcohol](https://mlsanity.com/paper/2309.08021/) (2023)
- [Creativity and Visual Communication from Machine to Musician: Sharing a Score through a Robotic Camera](https://mlsanity.com/paper/2409.05773/) (2024)
- [Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making](https://mlsanity.com/paper/2602.07668/) (2026)

## Abstract (arXiv)

To make safe transitions from autonomous to manual control, a vehicle must have a representation of the awareness of driver state; two metrics which quantify this state are the Observable Readiness Index and Takeover Time. In this work, we show that machine learning models which predict these two metrics are robust to multiple camera views, expanding from the limited view angles in prior research. Importantly, these models take as input feature vectors corresponding to hand location and activity as well as gaze location, and we explore the tradeoffs of different views in generating these feature vectors. Further, we introduce two metrics to evaluate the quality of control transitions following the takeover event (the maximal lateral deviation and velocity deviation) and compute correlations of these post-takeover metrics to the pre-takeover predictive metrics.