# Simultaneous Segmentation and Recognition: Towards more accurate Ego Gesture Recognition

arXiv: https://arxiv.org/abs/1909.08606 · HTML: https://mlsanity.com/paper/1909.08606/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 3 of 3)

- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition](https://mlsanity.com/paper/2110.15639/) (2021)
- [Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries](https://mlsanity.com/paper/2102.08065/) (2021)

## Abstract (arXiv)

Ego hand gestures can be used as an interface in AR and VR environments. While the context of an image is important for tasks like scene understanding, object recognition, image caption generation and activity recognition, it plays a minimal role in ego hand gesture recognition. An ego hand gesture used for AR and VR environments conveys the same information regardless of the background. With this idea in mind, we present our work on ego hand gesture recognition that produces embeddings from RBG images with ego hands, which are simultaneously used for ego hand segmentation and ego gesture recognition. To this extent, we achieved better recognition accuracy (96.9%) compared to the state of the art (92.2%) on the biggest ego hand gesture dataset available publicly. We present a gesture recognition deep neural network which recognises ego hand gestures from videos (videos containing a single gesture) by generating and recognising embeddings of ego hands from image sequences of varying length. We introduce the concept of simultaneous segmentation and recognition applied to ego hand gestures, present the network architecture, the training procedure and the results compared to the state of the art on the EgoGesture dataset