# DriverMHG: A Multi-Modal Dataset for Dynamic Recognition of Driver Micro Hand Gestures and a Real-Time Recognition Framework

arXiv: https://arxiv.org/abs/2003.00951 · HTML: https://mlsanity.com/paper/2003.00951/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 8 of 8)

- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Benchmarking Micro-action Recognition: Dataset, Methods, and Applications](https://mlsanity.com/paper/2403.05234/) (2024)
- [AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception](https://mlsanity.com/paper/2307.13933/) (2023)
- [Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours from Multimodal Videos](https://mlsanity.com/paper/2109.04021/) (2021)
- [Resource-Efficient Gesture Recognition using Low-Resolution Thermal Camera via Spiking Neural Networks and Sparse Segmentation](https://mlsanity.com/paper/2401.06563/) (2024)
- [A Deep Learning-based Multimodal Depth-Aware Dynamic Hand Gesture Recognition System](https://mlsanity.com/paper/2107.02543/) (2021)
- [Real-Time Driver Monitoring Systems through Modality and View Analysis](https://mlsanity.com/paper/2210.09441/) (2022)
- [Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates](https://mlsanity.com/paper/2204.04674/) (2022)

## Abstract (arXiv)

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded system. In this work, we propose an HCI system for dynamic recognition of driver micro hand gestures, which can have a crucial impact in automotive sector especially for safety related issues. For this purpose, we initially collected a dataset named Driver Micro Hand Gestures (DriverMHG), which consists of RGB, depth and infrared modalities. The challenges for dynamic recognition of micro hand gestures have been addressed by proposing a lightweight convolutional neural network (CNN) based architecture which operates online efficiently with a sliding window approach. For the CNN model, several 3-dimensional resource efficient networks are applied and their performances are analyzed. Online recognition of gestures has been performed with 3D-MobileNetV2, which provided the best offline accuracy among the applied networks with similar computational complexities. The final architecture is deployed on a driver simulator operating in real-time. We make DriverMHG dataset and our source code publicly available.