# Boosting Gesture Recognition with an Automatic Gesture Annotation Framework

arXiv: https://arxiv.org/abs/2401.11150 · HTML: https://mlsanity.com/paper/2401.11150/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 20 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Learning with Pseudo-Ensembles](https://mlsanity.com/paper/1412.4864/) (2014)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [A neural network based on SPD manifold learning for skeleton-based hand gesture recognition](https://mlsanity.com/paper/1904.12970/) (2019)
- [Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks](https://mlsanity.com/paper/1901.10323/) (2019)
- [Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention](https://mlsanity.com/paper/1907.08871/) (2019)
- [Enabling hand gesture customization on wrist-worn devices](https://mlsanity.com/paper/2203.15239/) (2022)
- [The Imaginative Generative Adversarial Network: Automatic Data Augmentation for Dynamic Skeleton-Based Hand Gesture and Human Action Recognition](https://mlsanity.com/paper/2105.13061/) (2021)
- [Low-latency hand gesture recognition with a low resolution thermal imager](https://mlsanity.com/paper/2004.11623/) (2020)

## Built on by (most cited first, 1 of 1)

- [Duo Streamers: A Streaming Gesture Recognition Framework](https://mlsanity.com/paper/2502.12297/) (2025)

## Abstract (arXiv)

Training a real-time gesture recognition model heavily relies on annotated data. However, manual data annotation is costly and demands substantial human effort. In order to address this challenge, we propose a framework that can automatically annotate gesture classes and identify their temporal ranges. Our framework consists of two key components: (1) a novel annotation model that leverages the Connectionist Temporal Classification (CTC) loss, and (2) a semi-supervised learning pipeline that enables the model to improve its performance by training on its own predictions, known as pseudo labels. These high-quality pseudo labels can also be used to enhance the accuracy of other downstream gesture recognition models. To evaluate our framework, we conducted experiments using two publicly available gesture datasets. Our ablation study demonstrates that our annotation model design surpasses the baseline in terms of both gesture classification accuracy (3-4% improvement) and localization accuracy (71-75% improvement). Additionally, we illustrate that the pseudo-labeled dataset produced from the proposed framework significantly boosts the accuracy of a pre-trained downstream gesture recognition model by 11-18%. We believe that this annotation framework has immense potential to improve the training of downstream gesture recognition models using unlabeled datasets.