mlsanityData to 7 Oct 2026

On the Utility of 3D Hand Poses for Action Recognition

arXiv 2403.09805 ↗First posted 14 Mar 2024cs.CV cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

3D hand pose is an underexplored modality for action recognition. Poses are compact yet informative and can greatly benefit applications with limited compute budgets. However, poses alone offer an incomplete understanding of actions, as they cannot fully capture objects and environments with which humans interact. We propose HandFormer, a novel multimodal transformer, to efficiently model hand-object interactions. HandFormer combines 3D hand poses at a high temporal resolution for fine-grained motion modeling with sparsely sampled RGB frames for encoding scene semantics. Observing the unique characteristics of hand poses, we temporally factorize hand modeling and represent each joint by its short-term trajectories. This factorized pose representation combined with sparse RGB samples is remarkably efficient and highly accurate. Unimodal HandFormer with only hand poses outperforms existing skeleton-based methods at 5x fewer FLOPs. With RGB, we achieve new state-of-the-art performance on Assembly101 and H2O with significant improvements in egocentric action recognition.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 52 of its 73 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

52 of 73 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
38 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  6. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  7. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  8. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  9. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  10. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  11. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  12. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  13. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  14. 2021Revisiting Skeleton-based Action Recognitioncited by 256
  15. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  16. 2019Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognitioncited by 228
  17. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  18. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  19. 2021Anticipative Video Transformercited by 197
  20. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  21. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  22. 2017Interpretable 3D Human Action Analysis with Temporal Convolutional Networkscited by 88
  23. 2020Spatial Temporal Transformer Network for Skeleton-based Action Recognitioncited by 76
  24. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  25. 2015P-CNN: Pose-based CNN Features for Action Recognitioncited by 58
  26. 2020VPN: Learning Video-Pose Embedding for Activities of Daily Livingcited by 57
  27. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 48
  28. 2022STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognitioncited by 43
  29. 2019SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognitioncited by 42
  30. 2021VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Livingcited by 35
  31. 2022Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videoscited by 35
  32. 2023On the Benefits of 3D Pose and Tracking for Human Action Recognitioncited by 32
  33. 2018Graph Edge Convolutional Neural Networks for Skeleton Based Action Recognitioncited by 28
  34. 2023Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognitioncited by 23
  35. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  36. 2021A Two-stream Neural Network for Pose-based Hand Gesture Recognitioncited by 10
  37. 2017Body Joint guided 3D Deep Convolutional Descriptors for Action Recognitioncited by 8
  38. 2024EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristbandcited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  2. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  3. 2025MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alpscited by 14
  4. 2025AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoningcited by 5
  5. 2024CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognitioncited by 3
  6. 2025EgoEvGesture: Gesture Recognition Based on Egocentric Event Cameracited by 3
  7. 2025Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognitioncited by 3
  8. 2025Cross-Modal Dual-Causal Learning for Long-Term Action Recognitioncited by 1
  9. 2025Efficient Egocentric Action Recognition with Multimodal Datacited by 0
  10. 2025Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challengescited by 0
  11. 2026BHaRNet: Reliability-Aware Body-Hand Modality Expertized Networks for Fine-grained Skeleton Action Recognitioncited by 0
  12. 2026Neuro-Symbolic Manipulation Understanding with Enriched Semantic Event Chainscited by 0
5 more
  1. 2026A Synthetic-Driven Vision System for Assembly Step Recognitioncited by 0
  2. 2026Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecastingcited by 0
  3. 2026STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognitioncited by 0
  4. 2026De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shiftcited by 0
  5. 2026ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wildcited by 0