# Unified Learning Approach for Egocentric Hand Gesture Recognition and Fingertip Detection

arXiv: https://arxiv.org/abs/2101.02047 · HTML: https://mlsanity.com/paper/2101.02047/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Analysis of the hands in egocentric vision: A survey](https://mlsanity.com/paper/1912.10867/) (2019)
- [Trajectory Aligned Features For First Person Action Recognition](https://mlsanity.com/paper/1604.02115/) (2016)
- [Rule Of Thumb: Deep derotation for improved fingertip detection](https://mlsanity.com/paper/1507.05726/) (2015)
- [A Real-time Hand Gesture Recognition and Human-Computer Interaction System](https://mlsanity.com/paper/1704.07296/) (2017)
- [Hand Action Detection from Ego-centric Depth Sequences with Error-correcting Hough Transform](https://mlsanity.com/paper/1606.02031/) (2016)
- [Hand Segmentation and Fingertip Tracking from Depth Camera Images Using Deep Convolutional Neural Network and Multi-task SegNet](https://mlsanity.com/paper/1901.03465/) (2019)
- [Fingertip in the Eye: A cascaded CNN pipeline for the real-time fingertip detection in egocentric videos](https://mlsanity.com/paper/1511.02282/) (2015)
- [AirPen: A Touchless Fingertip Based Gestural Interface for Smartphones and Head-Mounted Devices](https://mlsanity.com/paper/1904.06122/) (2019)

## Built on by (most cited first, 6 of 6)

- [Enhancing Diffusion Models with Text-Encoder Reinforcement Learning](https://mlsanity.com/paper/2311.15657/) (2023)
- [Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments](https://mlsanity.com/paper/2307.07757/) (2023)
- [Ultra-Range Gesture Recognition using a Web-Camera in Human-Robot Interaction](https://mlsanity.com/paper/2311.15361/) (2023)
- [3D Hand Pose and Shape Estimation from RGB Images for Keypoint-Based Hand Gesture Recognition](https://mlsanity.com/paper/2109.13879/) (2021)
- [Winning the CVPR'2022 AQTC Challenge: A Two-stage Function-centric Approach](https://mlsanity.com/paper/2206.09597/) (2022)
- [Study and Survey on Gesture Recognition Systems](https://mlsanity.com/paper/2312.00392/) (2023)

## Abstract (arXiv)

Head-mounted device-based human-computer interaction often requires egocentric recognition of hand gestures and fingertips detection. In this paper, a unified approach of egocentric hand gesture recognition and fingertip detection is introduced. The proposed algorithm uses a single convolutional neural network to predict the probabilities of finger class and positions of fingertips in one forward propagation. Instead of directly regressing the positions of fingertips from the fully connected layer, the ensemble of the position of fingertips is regressed from the fully convolutional network. Subsequently, the ensemble average is taken to regress the final position of fingertips. Since the whole pipeline uses a single network, it is significantly fast in computation. Experimental results show that the proposed method outperforms the existing fingertip detection approaches including the Direct Regression and the Heatmap-based framework. The effectiveness of the proposed method is also shown in-the-wild scenario as well as in a use-case of virtual reality.