mlsanityData to 7 Oct 2026

Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos

arXiv 2209.09484 ↗First posted 20 Sep 2022cs.CV cs.ROcited by 35

Not on the map. Why?

Abstract (arXiv)

Understanding dynamic hand motions and actions from egocentric RGB videos is a fundamental yet challenging task due to self-occlusion and ambiguity. To address occlusion and ambiguity, we develop a transformer-based framework to exploit temporal information for robust estimation. Noticing the different temporal granularity of and the semantic correlation between hand pose estimation and action recognition, we build a network hierarchy with two cascaded transformer encoders, where the first one exploits the short-term temporal cue for hand pose estimation, and the latter aggregates per-frame pose and object information over a longer time span to recognize the action. Our approach achieves competitive results on two first-person hand action benchmarks, namely FPHA and H2O. Extensive ablation studies verify our design choices.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 20 Sep 2022.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  12. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
30 more
  1. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  2. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  3. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  4. 2021Mesh Graphormercited by 264
  5. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  6. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  7. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  8. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  9. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  10. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  11. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  12. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  13. 2016Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognitioncited by 142
  14. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  15. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  16. 2017A New Representation of Skeleton Sequences for 3D Action Recognitioncited by 125
  17. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  18. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  19. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  20. 20182D/3D Pose Estimation and Action Recognition using Multitask Deep Learningcited by 103
  21. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  22. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 95
  23. 2017Interpretable 3D Human Action Analysis with Temporal Convolutional Networkscited by 88
  24. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  25. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  26. 2016Going Deeper into First-Person Activity Recognitioncited by 48
  27. 2021Forecasting Action through Contact Representations from First Person Videocited by 46
  28. 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36
  29. 2021RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Videocited by 23
  30. 2021Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memoriescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 12; 2025: 8; 2026: 9.

35 papers.

Sort:

By area:

  1. 2023An Outlook into the Future of Egocentric Visioncited by 84
  2. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  3. 2024Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimationcited by 19
  4. 2024On the Utility of 3D Hand Poses for Action Recognitioncited by 17
  5. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  6. 2024HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactionscited by 13
  7. 2024In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognitioncited by 12
  8. 2023A Hierarchical Graph-based Approach for Recognition and Description Generation of Bimanual Actions in Videoscited by 7
  9. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  10. 2024ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Imagescited by 4
  11. 2023Generative Hierarchical Temporal Transformer for Hand Pose and Action Modelingcited by 3
  12. 2024CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognitioncited by 3
23 more
  1. 2025EgoEvGesture: Gesture Recognition Based on Egocentric Event Cameracited by 3
  2. 2023Free-Form Composition Networks for Egocentric Action Recognitioncited by 2
  3. 2024SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptationcited by 2
  4. 2024SPiKE: 3D Human Pose from Point Cloud Sequencescited by 2
  5. 2024V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interactioncited by 2
  6. 2025Collaborative Learning for 3D Hand-Object Reconstruction and Compositional Action Recognition from Egocentric RGB Videos Using Superquadricscited by 2
  7. 2025Affordance-Guided Diffusion Prior for 3D Hand Reconstructioncited by 2
  8. 2026Towards Egocentric 3D Hand Pose Estimation in Unseen Domainscited by 2
  9. 2023CaSAR: Contact-aware Skeletal Action Recognitioncited by 1
  10. 2024SHARP: Segmentation of Hands and Arms by Range using Pseudo-Depth for Enhanced Egocentric 3D Hand Pose Estimation and Action Recognitioncited by 1
  11. 2025Visuo-Acoustic Hand Pose and Contact Estimationcited by 1
  12. 2025Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Imagescited by 1
  13. 2023ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertaintycited by 0
  14. 2025Multi Activity Sequence Alignment via Implicit Clusteringcited by 0
  15. 2025Person Identification from Egocentric Human-Object Interactions using 3D Hand Posecited by 0
  16. 2025DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactionscited by 0
  17. 2026Rethinking Graph Convolution for 2D-to-3D Hand Pose Liftingcited by 0
  18. 2026Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Trainingcited by 0
  19. 2026STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognitioncited by 0
  20. 2026EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Datasetcited by 0
  21. 2026De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shiftcited by 0
  22. 2026Functional Hand Type Prior for 3D Hand Pose Estimation and Action Recognition from Egocentric View Monocular Videoscited by 0
  23. 2026ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wildcited by 0