# Egocentric View Hand Action Recognition by Leveraging Hand Surface and Hand Grasp Type

arXiv: https://arxiv.org/abs/2109.03783 · HTML: https://mlsanity.com/paper/2109.03783/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 8 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input](https://mlsanity.com/paper/1610.04889/) (2016)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [Representation Flow for Action Recognition](https://mlsanity.com/paper/1810.01455/) (2018)
- [Trear: Transformer-based RGB-D Egocentric Action Recognition](https://mlsanity.com/paper/2101.03904/) (2021)
- [Curvature: A signature for Action Recognition in Video Sequences](https://mlsanity.com/paper/1904.13003/) (2019)

## Abstract (arXiv)

We introduce a multi-stage framework that uses mean curvature on a hand surface and focuses on learning interaction between hand and object by analyzing hand grasp type for hand action recognition in egocentric videos. The proposed method does not require 3D information of objects including 6D object poses which are difficult to annotate for learning an object's behavior while it interacts with hands. Instead, the framework synthesizes the mean curvature of the hand mesh model to encode the hand surface geometry in 3D space. Additionally, our method learns the hand grasp type which is highly correlated with the hand action. From our experiment, we notice that using hand grasp type and mean curvature of hand increases the performance of the hand action recognition.