mlsanityData to 7 Oct 2026

In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

arXiv 2404.09308 ↗First posted 14 Apr 2024cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Action recognition is essential for egocentric video understanding, allowing automatic and continuous monitoring of Activities of Daily Living (ADLs) without user effort. Existing literature focuses on 3D hand pose input, which requires computationally intensive depth estimation networks or wearing an uncomfortable depth sensor. In contrast, there has been insufficient research in understanding 2D hand pose for egocentric action recognition, despite the availability of user-friendly smart glasses in the market capable of capturing a single RGB image. Our study aims to fill this research gap by exploring the field of 2D hand pose estimation for egocentric action recognition, making two contributions. Firstly, we introduce two novel approaches for 2D hand pose estimation, namely EffHandNet for single-hand estimation and EffHandEgoNet, tailored for an egocentric perspective, capturing interactions between hands and objects. Both methods outperform state-of-the-art models on H2O and FPHA public benchmarks. Secondly, we present a robust action recognition architecture from 2D hand and object poses. This method incorporates EffHandEgoNet, and a transformer-based action recognition method. Evaluated on H2O and FPHA datasets, our architecture has a faster inference time and achieves an accuracy of 91.32% and 94.43%, respectively, surpassing state of the art, including 3D-based methods. Our work demonstrates that using 2D skeletal data is a robust approach for egocentric action understanding. Extensive evaluation and ablation studies show the impact of the hand pose estimation approach, and how each input affects the overall performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

30 of 47 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  10. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  11. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  12. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
18 more
  1. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  3. 2018Albumentations: fast and flexible image augmentationscited by 558
  4. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  5. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  6. 2021Revisiting Skeleton-based Action Recognitioncited by 256
  7. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  8. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  9. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  11. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  12. 2023MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Networkcited by 62
  13. 2019A neural network based on SPD manifold learning for skeleton-based hand gesture recognitioncited by 46
  14. 2018Can 3D Pose be Learned from 2D Projections Alone?cited by 44
  15. 2022Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videoscited by 35
  16. 2019Analysis of the hands in egocentric vision: A surveycited by 25
  17. 2020Attention! A Lightweight 2D Hand Pose Estimation Approachcited by 9
  18. 2021Domain and View-point Agnostic Hand Action Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 5.

12 papers.

  1. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  2. 2025Survey on Hand Gesture Recognition from Visual Inputcited by 10
  3. 2025EgoEvGesture: Gesture Recognition Based on Egocentric Event Cameracited by 3
  4. 2026Towards Egocentric 3D Hand Pose Estimation in Unseen Domainscited by 2
  5. 2026EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Camerascited by 2
  6. 2024SHARP: Segmentation of Hands and Arms by Range using Pseudo-Depth for Enhanced Egocentric 3D Hand Pose Estimation and Action Recognitioncited by 1
  7. 2025ACT360: An Efficient 360-Degree Action Detection and Summarization Framework for Mission-Critical Training and Debriefingcited by 1
  8. 2024REST-HANDS: Rehabilitation with Egocentric Vision Using Smartglasses for Treatment of Hands after Surviving Strokecited by 0
  9. 2025Person Identification from Egocentric Human-Object Interactions using 3D Hand Posecited by 0
  10. 2026STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognitioncited by 0
  11. 2026EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Datasetcited by 0
  12. 2026De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shiftcited by 0