# Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN

arXiv: https://arxiv.org/abs/2406.15003 · HTML: https://mlsanity.com/paper/2406.15003/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 78 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1804.07453/) (2018)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [A neural network based on SPD manifold learning for skeleton-based hand gesture recognition](https://mlsanity.com/paper/1904.12970/) (2019)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training](https://mlsanity.com/paper/1812.06145/) (2018)
- [Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention](https://mlsanity.com/paper/1907.08871/) (2019)
- [Make Skeleton-based Action Recognition Model Smaller, Faster and Better](https://mlsanity.com/paper/1907.09658/) (2019)
- [IPN Hand: A Video Dataset and Benchmark for Real-Time Continuous Hand Gesture Recognition](https://mlsanity.com/paper/2005.02134/) (2020)
- [A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding](https://mlsanity.com/paper/1807.00676/) (2018)
- [Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2007.03263/) (2020)
- [CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2007.11983/) (2020)
- [Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition](https://mlsanity.com/paper/1804.07187/) (2018)
- [DeepGRU: Deep Gesture Recognition Utility](https://mlsanity.com/paper/1810.12514/) (2018)
- [PAN: Towards Fast Action Recognition via Learning Persistence of Appearance](https://mlsanity.com/paper/2008.03462/) (2020)
- [Motion Feature Augmented Recurrent Neural Network for Skeleton-based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/1708.03278/) (2017)
- [Multi-modal Fusion for Single-Stage Continuous Gesture Recognition](https://mlsanity.com/paper/2011.04945/) (2020)
- [A Two-stream Neural Network for Pose-based Hand Gesture Recognition](https://mlsanity.com/paper/2101.08926/) (2021)
- [Dynamic Gesture Recognition by Using CNNs and Star RGB: a Temporal Information Condensation](https://mlsanity.com/paper/1904.08505/) (2019)
- [CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition](https://mlsanity.com/paper/2004.09215/) (2020)
- [HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based Gesture Recognition](https://mlsanity.com/paper/2106.13391/) (2021)
- [Skeleton-Based Hand Gesture Recognition by Learning SPD Matrices with Neural Networks](https://mlsanity.com/paper/1905.07917/) (2019)
- [Domain and View-point Agnostic Hand Action Recognition](https://mlsanity.com/paper/2103.02303/) (2021)
- [A Deep Learning-based Multimodal Depth-Aware Dynamic Hand Gesture Recognition System](https://mlsanity.com/paper/2107.02543/) (2021)
- [An Ensemble of Knowledge Sharing Models for Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2008.05732/) (2020)
- [A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2401.00409/) (2023)
- [3D dynamic hand gestures recognition using the Leap Motion sensor and convolutional neural networks](https://mlsanity.com/paper/2003.01450/) (2020)
- [Skeleton-based Hand-Gesture Recognition with Lightweight Graph Convolutional Networks](https://mlsanity.com/paper/2104.04255/) (2021)

## Built on by (most cited first, 2 of 2)

- [Computer Vision-Driven Gesture Recognition: Toward Natural and Intuitive Human-Computer](https://mlsanity.com/paper/2412.18321/) (2024)
- [Duo Streamers: A Streaming Gesture Recognition Framework](https://mlsanity.com/paper/2502.12297/) (2025)

## Abstract (arXiv)

Hand Gesture Recognition (HGR) enables intuitive human-computer interactions in various real-world contexts. However, existing frameworks often struggle to meet the real-time requirements essential for practical HGR applications. This study introduces a robust, skeleton-based framework for dynamic HGR that simplifies the recognition of dynamic hand gestures into a static image classification task, effectively reducing both hardware and computational demands. Our framework utilizes a data-level fusion technique to encode 3D skeleton data from dynamic gestures into static RGB spatiotemporal images. It incorporates a specialized end-to-end Ensemble Tuner (e2eET) Multi-Stream CNN architecture that optimizes the semantic connections between data representations while minimizing computational needs. Tested across five benchmark datasets (SHREC'17, DHG-14/28, FPHA, LMDHG, and CNR), the framework showed competitive performance with the state-of-the-art. Its capability to support real-time HGR applications was also demonstrated through deployment on standard consumer PC hardware, showcasing low latency and minimal resource usage in real-world settings. The successful deployment of this framework underscores its potential to enhance real-time applications in fields such as virtual/augmented reality, ambient intelligence, and assistive technologies, providing a scalable and efficient solution for dynamic gesture recognition.