# Head and eye egocentric gesture recognition for human-robot interaction using eyewear cameras

arXiv: https://arxiv.org/abs/2201.11500 · HTML: https://mlsanity.com/paper/2201.11500/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 47 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 7 of 7)

- [Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model](https://mlsanity.com/paper/1709.03966/) (2017)
- [Content-Aware Unsupervised Deep Homography Estimation](https://mlsanity.com/paper/1909.05983/) (2019)
- [Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction](https://mlsanity.com/paper/1405.0006/) (2014)
- [Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks](https://mlsanity.com/paper/1604.05633/) (2016)
- [Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language](https://mlsanity.com/paper/2103.08882/) (2021)
- [Gesture-based Human-Machine Interaction: Taxonomy, Problem Definition, and Analysis](https://mlsanity.com/paper/2201.10186/) (2022)
- [Rebellion and Obedience: The Effects of Intention Prediction in Cooperative Handheld Robots](https://mlsanity.com/paper/1903.08158/) (2019)

## Built on by (most cited first, 3 of 3)

- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [An Egocentric Vision-Language Model based Portable Real-time Smart Assistant](https://mlsanity.com/paper/2503.04250/) (2025)
- [Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision](https://mlsanity.com/paper/2503.06089/) (2025)

## Abstract (arXiv)

Non-verbal communication plays a particularly important role in a wide range of scenarios in Human-Robot Interaction (HRI). Accordingly, this work addresses the problem of human gesture recognition. In particular, we focus on head and eye gestures, and adopt an egocentric (first-person) perspective using eyewear cameras. We argue that this egocentric view may offer a number of conceptual and technical benefits over scene- or robot-centric perspectives. A motion-based recognition approach is proposed, which operates at two temporal granularities. Locally, frame-to-frame homographies are estimated with a convolutional neural network (CNN). The output of this CNN is input to a long short-term memory (LSTM) to capture longer-term temporal visual relationships, which are relevant to characterize gestures. Regarding the configuration of the network architecture, one particularly interesting finding is that using the output of an internal layer of the homography CNN increases the recognition rate with respect to using the homography matrix itself. While this work focuses on action recognition, and no robot or user study has been conducted yet, the system has been designed to meet real-time constraints. The encouraging results suggest that the proposed egocentric perspective is viable, and this proof-of-concept work provides novel and useful contributions to the exciting area of HRI.