# Looking at Hands in Autonomous Vehicles: A ConvNet Approach using Part Affinity Fields

arXiv: https://arxiv.org/abs/1804.01176 · HTML: https://mlsanity.com/paper/1804.01176/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2018.

## Built on by (most cited first, 7 of 7)

- [LaneAF: Robust Multi-Lane Detection with Affinity Fields](https://mlsanity.com/paper/2103.12040/) (2021)
- [Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data](https://mlsanity.com/paper/2104.11489/) (2021)
- [Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction](https://mlsanity.com/paper/2301.05805/) (2023)
- [Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation](https://mlsanity.com/paper/2107.12932/) (2021)
- [(Safe) SMART Hands: Hand Activity Analysis and Distraction Alerts Using a Multi-Camera Framework](https://mlsanity.com/paper/2301.05838/) (2023)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)
- [Integrated In-vehicle Monitoring System Using 3D Human Pose Estimation and Seat Belt Segmentation](https://mlsanity.com/paper/2204.07946/) (2022)

## Abstract (arXiv)

In the context of autonomous driving, where humans may need to take over in the event where the computer may issue a takeover request, a key step towards driving safety is the monitoring of the hands to ensure the driver is ready for such a request. This work, focuses on the first step of this process, which is to locate the hands. Such a system must work in real-time and under varying harsh lighting conditions. This paper introduces a fast ConvNet approach, based on the work of original work of OpenPose for full body joint estimation. The network is modified with fewer parameters and retrained using our own day-time naturalistic autonomous driving dataset to estimate joint and affinity heatmaps for driver & passenger's wrist and elbows, for a total of 8 joint classes and part affinity fields between each wrist-elbow pair. The approach runs real-time on real-world data at 40 fps on multiple drivers and passengers. The system is extensively evaluated both quantitatively and qualitatively, showing at least 95% detection performance on joint localization and arm-angle estimation.