# Egocentric Gesture Recognition for Head-Mounted AR devices

arXiv: https://arxiv.org/abs/1808.05380 · HTML: https://mlsanity.com/paper/1808.05380/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2018.

## Built on by (most cited first, 6 of 6)

- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation](https://mlsanity.com/paper/2406.18070/) (2024)
- [Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control](https://mlsanity.com/paper/2104.03893/) (2021)
- [ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction](https://mlsanity.com/paper/2504.01472/) (2025)
- [Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision](https://mlsanity.com/paper/2503.06089/) (2025)
- [x-RAGE: eXtended Reality -- Action & Gesture Events Dataset](https://mlsanity.com/paper/2410.19486/) (2024)

## Abstract (arXiv)

Natural interaction with virtual objects in AR/VR environments makes for a smooth user experience. Gestures are a natural extension from real world to augmented space to achieve these interactions. Finding discriminating spatio-temporal features relevant to gestures and hands in ego-view is the primary challenge for recognising egocentric gestures. In this work we propose a data driven end-to-end deep learning approach to address the problem of egocentric gesture recognition, which combines an ego-hand encoder network to find ego-hand features, and a recurrent neural network to discern temporally discriminating features. Since deep learning networks are data intensive, we propose a novel data augmentation technique using green screen capture to alleviate the problem of ground truth annotation. In addition we publish a dataset of 10 gestures performed in a natural fashion in front of a green screen for training and the same 10 gestures performed in different natural scenes without green screen for validation. We also present the results of our network's performance in comparison to the state-of-the-art using the AirGest dataset