# Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1901.10323 · HTML: https://mlsanity.com/paper/1901.10323/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 30 of 30)

- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [BM-NAS: Bilevel Multimodal Neural Architecture Search](https://mlsanity.com/paper/2104.09379/) (2021)
- [Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition](https://mlsanity.com/paper/2112.09129/) (2021)
- [Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition](https://mlsanity.com/paper/2102.05348/) (2021)
- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition](https://mlsanity.com/paper/2211.09146/) (2022)
- [Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition](https://mlsanity.com/paper/2110.15639/) (2021)
- [GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2405.11180/) (2024)
- [PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors](https://mlsanity.com/paper/2304.05440/) (2023)
- [Resource-Efficient Gesture Recognition using Low-Resolution Thermal Camera via Spiking Neural Networks and Sparse Segmentation](https://mlsanity.com/paper/2401.06563/) (2024)
- [HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2412.01508/) (2024)
- [It's all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight Camera](https://mlsanity.com/paper/2310.01659/) (2023)
- [MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2409.03890/) (2024)
- [ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2411.07118/) (2024)
- [Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2501.00935/) (2025)
- [Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay](https://mlsanity.com/paper/2505.20049/) (2025)
- [Hand gesture detection in tests performed by older adults](https://mlsanity.com/paper/2110.14461/) (2021)
- [AI-enabled Sound Pattern Recognition on Asthma Medication Adherence: Evaluation with the RDA Benchmark Suite](https://mlsanity.com/paper/2205.15360/) (2022)
- [Long-Distance Gesture Recognition using Dynamic Neural Networks](https://mlsanity.com/paper/2308.04643/) (2023)
- [Boosting Gesture Recognition with an Automatic Gesture Annotation Framework](https://mlsanity.com/paper/2401.11150/) (2024)
- [Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables](https://mlsanity.com/paper/2503.07825/) (2025)
- [Efficient Gesture Recognition for the Assistance of Visually Impaired People using Multi-Head Neural Networks](https://mlsanity.com/paper/2205.06980/) (2022)
- [SynthoGestures: A Novel Framework for Synthetic Dynamic Hand Gesture Generation for Driving Scenarios](https://mlsanity.com/paper/2309.04421/) (2023)
- [Duo Streamers: A Streaming Gesture Recognition Framework](https://mlsanity.com/paper/2502.12297/) (2025)
- [Accurate online action and gesture recognition system using detectors and Deep SPD Siamese Networks](https://mlsanity.com/paper/2511.05250/) (2025)
- [OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognition](https://mlsanity.com/paper/2512.16727/) (2025)
- [Online Hand Gesture Recognition Using 3D Convolutional Neural Networks](https://mlsanity.com/paper/2605.23409/) (2026)
- [Recognizing Co-Speech Gestures in-the-Wild](https://mlsanity.com/paper/2605.31589/) (2026)
- [UniCon-Former: Unified Convolution Transformer is All You Need for Hand Gesture Recognition](https://mlsanity.com/paper/2608.13217/) (2026)
- [Frame-Synchronous Hand Gesture Detection by Projected Winding Order](https://mlsanity.com/paper/2609.13269/) (2026)

## Abstract (arXiv)

Real-time recognition of dynamic hand gestures from video streams is a challenging task since (i) there is no indication when a gesture starts and ends in the video, (ii) performed gestures should only be recognized once, and (iii) the entire architecture should be designed considering the memory and power budget. In this work, we address these challenges by proposing a hierarchical structure enabling offline-working convolutional neural network (CNN) architectures to operate online efficiently by using sliding window approach. The proposed architecture consists of two models: (1) A detector which is a lightweight CNN architecture to detect gestures and (2) a classifier which is a deep CNN to classify the detected gestures. In order to evaluate the single-time activations of the detected gestures, we propose to use Levenshtein distance as an evaluation metric since it can measure misclassifications, multiple detections, and missing detections at the same time. We evaluate our architecture on two publicly available datasets - EgoGesture and NVIDIA Dynamic Hand Gesture Datasets - which require temporal detection and classification of the performed hand gestures. ResNeXt-101 model, which is used as a classifier, achieves the state-of-the-art offline classification accuracy of 94.04% and 83.82% for depth modality on EgoGesture and NVIDIA benchmarks, respectively. In real-time detection and classification, we obtain considerable early detections while achieving performances close to offline operation. The codes and pretrained models used in this work are publicly available.