# Towards Open-World Gesture Recognition

arXiv: https://arxiv.org/abs/2401.11144 · HTML: https://mlsanity.com/paper/2401.11144/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 75 references that are in the map.
- First posted on arXiv 20 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions](https://mlsanity.com/paper/1910.10261/) (2019)
- [Rethinking Experience Replay: a Bag of Tricks for Continual Learning](https://mlsanity.com/paper/2010.05595/) (2020)
- [Enabling hand gesture customization on wrist-worn devices](https://mlsanity.com/paper/2203.15239/) (2022)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [The Imaginative Generative Adversarial Network: Automatic Data Augmentation for Dynamic Skeleton-Based Hand Gesture and Human Action Recognition](https://mlsanity.com/paper/2105.13061/) (2021)
- [Hand tracking for immersive virtual reality: opportunities and challenges](https://mlsanity.com/paper/2103.14853/) (2021)
- [CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition](https://mlsanity.com/paper/2004.09215/) (2020)

## Built on by (most cited first, 3 of 3)

- [Duo Streamers: A Streaming Gesture Recognition Framework](https://mlsanity.com/paper/2502.12297/) (2025)
- [AutoMR: A Universal Time Series Motion Recognition Pipeline](https://mlsanity.com/paper/2502.15228/) (2025)
- [CLANE: Continual Learning of Actions on Neuromorphic Hardware from Event Cameras](https://mlsanity.com/paper/2605.28387/) (2026)

## Abstract (arXiv)

Providing users with accurate gestural interfaces, such as gesture recognition based on wrist-worn devices, is a key challenge in mixed reality. However, static machine learning processes in gesture recognition assume that training and test data come from the same underlying distribution. Unfortunately, in real-world applications involving gesture recognition, such as gesture recognition based on wrist-worn devices, the data distribution may change over time. We formulate this problem of adapting recognition models to new tasks, where new data patterns emerge, as open-world gesture recognition (OWGR). We propose the use of continual learning to enable machine learning models to be adaptive to new tasks without degrading performance on previously learned tasks. However, the process of exploring parameters for questions around when, and how, to train and deploy recognition models requires resource-intensive user studies may be impractical. To address this challenge, we propose a design engineering approach that enables offline analysis on a collected large-scale dataset by systematically examining various parameters and comparing different continual learning methods. Finally, we provide design guidelines to enhance the development of an open-world wrist-worn gesture recognition process.