# IPN Hand: A Video Dataset and Benchmark for Real-Time Continuous Hand Gesture Recognition

arXiv: https://arxiv.org/abs/2005.02134 · HTML: https://mlsanity.com/paper/2005.02134/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2020.

## Built on by (most cited first, 22 of 22)

- [HaGRID - HAnd Gesture Recognition Image Dataset](https://mlsanity.com/paper/2206.08219/) (2022)
- [Privacy-Preserving Action Recognition via Motion Difference Quantization](https://mlsanity.com/paper/2208.02459/) (2022)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [SocialGesture: Delving into Multi-person Gesture Understanding](https://mlsanity.com/paper/2504.02244/) (2025)
- [Selective, Interpretable, and Motion Consistent Privacy Attribute Obfuscation for Action Recognition](https://mlsanity.com/paper/2403.12710/) (2024)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [Co-Speech Gesture Detection through Multi-Phase Sequence Labeling](https://mlsanity.com/paper/2308.10680/) (2023)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation](https://mlsanity.com/paper/2409.10535/) (2024)
- [Omni-MMSI: Toward Identity-attributed Social Interaction Understanding](https://mlsanity.com/paper/2604.00267/) (2026)
- [HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2412.01508/) (2024)
- [edgeVLM: Cloud-edge Collaborative Real-time VLM based on Context Transfer](https://mlsanity.com/paper/2508.12638/) (2025)
- [Leveraging Speech for Gesture Detection in Multimodal Communication](https://mlsanity.com/paper/2404.14952/) (2024)
- [Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN](https://mlsanity.com/paper/2406.15003/) (2024)
- [MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives](https://mlsanity.com/paper/2501.04184/) (2025)
- [EHWGesture -- A dataset for multimodal understanding of clinical gestures](https://mlsanity.com/paper/2509.07525/) (2025)
- [Hand gesture detection in tests performed by older adults](https://mlsanity.com/paper/2110.14461/) (2021)
- [Challenges of the Creation of a Dataset for Vision Based Human Hand Action Recognition in Industrial Assembly](https://mlsanity.com/paper/2303.03716/) (2023)
- [AirLetters: An Open Video Dataset of Characters Drawn in the Air](https://mlsanity.com/paper/2410.02921/) (2024)
- [Gestura: A LVLM-Powered System Bridging Motion and Semantics for Real-Time Free-Form Gesture Understanding](https://mlsanity.com/paper/2510.21814/) (2025)
- [Unsafe2Safe: Controllable Image Anonymization for Downstream Utility](https://mlsanity.com/paper/2603.28605/) (2026)
- [A multimodal gesture recognition dataset for desktop human-computer interaction](https://mlsanity.com/paper/2401.03828/) (2024)

## Abstract (arXiv)

In this paper, we introduce a new benchmark dataset named IPN Hand with sufficient size, variety, and real-world elements able to train and evaluate deep neural networks. This dataset contains more than 4,000 gesture samples and 800,000 RGB frames from 50 distinct subjects. We design 13 different static and dynamic gestures focused on interaction with touchless screens. We especially consider the scenario when continuous gestures are performed without transition states, and when subjects perform natural movements with their hands as non-gesture actions. Gestures were collected from about 30 diverse scenes, with real-world variation in background and illumination. With our dataset, the performance of three 3D-CNN models is evaluated on the tasks of isolated and continuous real-time HGR. Furthermore, we analyze the possibility of increasing the recognition accuracy by adding multiple modalities derived from RGB frames, i.e., optical flow and semantic segmentation, while keeping the real-time performance of the 3D-CNN model. Our empirical study also provides a comparison with the publicly available nvGesture (NVIDIA) dataset. The experimental results show that the state-of-the-art ResNext-101 model decreases about 30% accuracy when using our real-world dataset, demonstrating that the IPN Hand dataset can be used as a benchmark, and may help the community to step forward in the continuous HGR. Our dataset and pre-trained models used in the evaluation are publicly available at https://github.com/GibranBenitez/IPN-hand.