# Enabling hand gesture customization on wrist-worn devices

arXiv: https://arxiv.org/abs/2203.15239 · HTML: https://mlsanity.com/paper/2203.15239/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 102 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [Delta-encoder: an effective sample synthesis method for few-shot object recognition](https://mlsanity.com/paper/1806.04734/) (2018)
- [Online Few-shot Gesture Learning on a Neuromorphic Processor](https://mlsanity.com/paper/2008.01151/) (2020)

## Built on by (most cited first, 15 of 15)

- [Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://mlsanity.com/paper/2310.04621/) (2023)
- [LipLearner: Customizable Silent Speech Interactions on Mobile Devices](https://mlsanity.com/paper/2302.05907/) (2023)
- [Time2Stop: Adaptive and Explainable Human-AI Loop for Smartphone Overuse Intervention](https://mlsanity.com/paper/2403.05584/) (2024)
- [GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents](https://mlsanity.com/paper/2310.12821/) (2023)
- [WatchGuardian: Enabling User-Defined Personalized Just-in-Time Intervention on Smartwatch](https://mlsanity.com/paper/2502.05783/) (2025)
- [Towards Open-World Gesture Recognition](https://mlsanity.com/paper/2401.11144/) (2024)
- [Advancing Location-Invariant and Device-Agnostic Motion Activity Recognition on Wearable Devices](https://mlsanity.com/paper/2402.03714/) (2024)
- [SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions](https://mlsanity.com/paper/2603.19529/) (2026)
- [Boosting Gesture Recognition with an Automatic Gesture Annotation Framework](https://mlsanity.com/paper/2401.11150/) (2024)
- [Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation](https://mlsanity.com/paper/2405.08969/) (2024)
- [TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition](https://mlsanity.com/paper/2608.15861/) (2026)
- [Fast and Expressive Gesture Recognition using a Combination-Homomorphic Electromyogram Encoder](https://mlsanity.com/paper/2311.14675/) (2023)
- [Single-tap Latency Reduction with Single- or Double- tap Prediction](https://mlsanity.com/paper/2408.02525/) (2024)
- [PoseAugment: Generative Human Pose Data Augmentation with Physical Plausibility for IMU-based Motion Capture](https://mlsanity.com/paper/2409.14101/) (2024)
- [BadminSense: Enabling Fine-Grained Badminton Stroke Evaluation on a Single Smartwatch](https://mlsanity.com/paper/2603.21825/) (2026)

## Abstract (arXiv)

We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an accelerometer-gyroscope recognition model with a cross-user accuracy of 95.7% and a false-positive rate of 0.6 per hour when tested on everyday non-gesture data. Next, we design a few-shot learning framework which derives a lightweight model from our pre-trained model, enabling knowledge transfer without performance degradation. We validate our approach through a user study (N=20) examining on-device customization from 12 new gestures, resulting in an average accuracy of 55.3%, 83.1%, and 87.2% on using one, three, or five shots when adding a new gesture, while maintaining the same recognition accuracy and false-positive rate from the pre-existing gesture set. We further evaluate the usability of our real-time implementation with a user experience study (N=20). Our results highlight the effectiveness, learnability, and usability of our customization framework. Our approach paves the way for a future where users are no longer bound to pre-existing gestures, freeing them to creatively introduce new gestures tailored to their preferences and abilities.