# A Deep Learning-based Multimodal Depth-Aware Dynamic Hand Gesture Recognition System

arXiv: https://arxiv.org/abs/2107.02543 · HTML: https://mlsanity.com/paper/2107.02543/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 52 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Hidden Two-Stream Convolutional Networks for Action Recognition](https://mlsanity.com/paper/1704.00389/) (2017)
- [Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention](https://mlsanity.com/paper/1907.08871/) (2019)
- [CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2007.11983/) (2020)
- [Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition](https://mlsanity.com/paper/1804.07187/) (2018)
- [DriverMHG: A Multi-Modal Dataset for Dynamic Recognition of Driver Micro Hand Gestures and a Real-Time Recognition Framework](https://mlsanity.com/paper/2003.00951/) (2020)
- [Low-latency hand gesture recognition with a low resolution thermal imager](https://mlsanity.com/paper/2004.11623/) (2020)
- [Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences](https://mlsanity.com/paper/1501.04686/) (2015)
- [Hand-tremor frequency estimation in videos](https://mlsanity.com/paper/1809.03218/) (2018)
- [An Ensemble of Knowledge Sharing Models for Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2008.05732/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [BdSLW60: A Word-Level Bangla Sign Language Dataset](https://mlsanity.com/paper/2402.08635/) (2024)
- [HaSPeR: An Image Repository for Hand Shadow Puppet Recognition](https://mlsanity.com/paper/2408.10360/) (2024)
- [Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN](https://mlsanity.com/paper/2406.15003/) (2024)
- [Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Tasks](https://mlsanity.com/paper/2506.04367/) (2025)

## Abstract (arXiv)

The dynamic hand gesture recognition task has seen studies on various unimodal and multimodal methods. Previously, researchers have explored depth and 2D-skeleton-based multimodal fusion CRNNs (Convolutional Recurrent Neural Networks) but have had limitations in getting expected recognition results. In this paper, we revisit this approach to hand gesture recognition and suggest several improvements. We observe that raw depth images possess low contrast in the hand regions of interest (ROI). They do not highlight important fine details, such as finger orientation, overlap between the finger and palm, or overlap between multiple fingers. We thus propose quantizing the depth values into several discrete regions, to create a higher contrast between several key parts of the hand. In addition, we suggest several ways to tackle the high variance problem in existing multimodal fusion CRNN architectures. We evaluate our method on two benchmarks: the DHG-14/28 dataset and the SHREC'17 track dataset. Our approach shows a significant improvement in accuracy and parameter efficiency over previous similar multimodal methods, with a comparable result to the state-of-the-art.