# HandyNet: A One-stop Solution to Detect, Segment, Localize & Analyze Driver Hands

arXiv: https://arxiv.org/abs/1804.07834 · HTML: https://mlsanity.com/paper/1804.07834/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2018.

## Built on by (most cited first, 6 of 6)

- [Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction](https://mlsanity.com/paper/2301.05805/) (2023)
- [Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation](https://mlsanity.com/paper/2107.12932/) (2021)
- [Multi-View Video-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2109.11747/) (2021)
- [(Safe) SMART Hands: Hand Activity Analysis and Distraction Alerts Using a Multi-Camera Framework](https://mlsanity.com/paper/2301.05838/) (2023)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)
- [Hands-Up: Leveraging Synthetic Data for Hands-On-Wheel Detection](https://mlsanity.com/paper/2206.00148/) (2022)

## Abstract (arXiv)

Tasks related to human hands have long been part of the computer vision community. Hands being the primary actuators for humans, convey a lot about activities and intents, in addition to being an alternative form of communication/interaction with other humans and machines. In this study, we focus on training a single feedforward convolutional neural network (CNN) capable of executing many hand related tasks that may be of use in autonomous and semi-autonomous vehicles of the future. The resulting network, which we refer to as HandyNet, is capable of detecting, segmenting and localizing (in 3D) driver hands inside a vehicle cabin. The network is additionally trained to identify handheld objects that the driver may be interacting with. To meet the data requirements to train such a network, we propose a method for cheap annotation based on chroma-keying, thereby bypassing weeks of human effort required to label such data. This process can generate thousands of labeled training samples in an efficient manner, and may be replicated in new environments with relative ease.