# The Imaginative Generative Adversarial Network: Automatic Data Augmentation for Dynamic Skeleton-Based Hand Gesture and Human Action Recognition

arXiv: https://arxiv.org/abs/2105.13061 · HTML: https://mlsanity.com/paper/2105.13061/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 27 of 27)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [Long Short-Term Memory-Networks for Machine Reading](https://mlsanity.com/paper/1601.06733/) (2016)
- [Synthetic Data Augmentation using GAN for Improved Liver Lesion Classification](https://mlsanity.com/paper/1801.02385/) (2018)
- [Unsupervised Cross-Domain Image Generation](https://mlsanity.com/paper/1611.02200/) (2016)
- [Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation](https://mlsanity.com/paper/1906.02611/) (2019)
- [A Bayesian Data Augmentation Approach for Learning Deep Models](https://mlsanity.com/paper/1710.10564/) (2017)
- [Affinity and Diversity: Quantifying Mechanisms of Data Augmentation](https://mlsanity.com/paper/2002.08973/) (2020)
- [Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation](https://mlsanity.com/paper/1805.09707/) (2018)
- [APAC: Augmented PAttern Classification with Neural Networks](https://mlsanity.com/paper/1505.03229/) (2015)
- [Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention](https://mlsanity.com/paper/1907.08871/) (2019)
- [Make Skeleton-based Action Recognition Model Smaller, Faster and Better](https://mlsanity.com/paper/1907.09658/) (2019)
- [Quo Vadis, Skeleton Action Recognition ?](https://mlsanity.com/paper/2007.02072/) (2020)
- [Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2007.03263/) (2020)
- [CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2007.11983/) (2020)
- [Investigation of Different Skeleton Features for CNN-based 3D Action Recognition](https://mlsanity.com/paper/1705.00835/) (2017)
- [Skeleton-based Action Recognition Using LSTM and CNN](https://mlsanity.com/paper/1707.02356/) (2017)
- [Leveraging GANs to Improve Continuous Path Keyboard Input Models](https://mlsanity.com/paper/2004.07800/) (2020)

## Built on by (most cited first, 8 of 8)

- [Towards Open-World Gesture Recognition](https://mlsanity.com/paper/2401.11144/) (2024)
- [Part-aware Prototypical Graph Network for One-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2208.09150/) (2022)
- [Boosting Gesture Recognition with an Automatic Gesture Annotation Framework](https://mlsanity.com/paper/2401.11150/) (2024)
- [Gesture2Text: A Generalizable Decoder for Word-Gesture Keyboards in XR Through Trajectory Coarse Discretization and Pre-training](https://mlsanity.com/paper/2410.18099/) (2024)
- [A Diffusion-based Data Generator for Training Object Recognition Models in Ultra-Range Distance](https://mlsanity.com/paper/2404.09846/) (2024)
- [Duo Streamers: A Streaming Gesture Recognition Framework](https://mlsanity.com/paper/2502.12297/) (2025)
- [AutoMR: A Universal Time Series Motion Recognition Pipeline](https://mlsanity.com/paper/2502.15228/) (2025)
- [Generative Data Augmentation for Skeleton Action Recognition](https://mlsanity.com/paper/2604.14933/) (2026)

## Abstract (arXiv)

Deep learning approaches deliver state-of-the-art performance in recognition of spatiotemporal human motion data. However, one of the main challenges in these recognition tasks is limited available training data. Insufficient training data results in over-fitting and data augmentation is one approach to address this challenge. Existing data augmentation strategies based on scaling, shifting and interpolating offer limited generalizability and typically require detailed inspection of the dataset as well as hundreds of GPU hours for hyperparameter optimization. In this paper, we present a novel automatic data augmentation model, the Imaginative Generative Adversarial Network (GAN), that approximates the distribution of the input data and samples new data from this distribution. It is automatic in that it requires no data inspection and little hyperparameter tuning and therefore it is a low-cost and low-effort approach to generate synthetic data. We demonstrate our approach on small-scale skeleton-based datasets with a comprehensive experimental analysis. Our results show that the augmentation strategy is fast to train and can improve classification accuracy for both conventional neural networks and state-of-the-art methods.