mlsanityData to 7 Oct 2026

Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation

arXiv 2203.16202 ↗First posted 30 Mar 2022cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose an approach to estimate arm and hand dynamics from monocular video by utilizing the relationship between arm and hand. Although monocular full human motion capture technologies have made great progress in recent years, recovering accurate and plausible arm twists and hand gestures from in-the-wild videos still remains a challenge. To solve this problem, our solution is proposed based on the fact that arm poses and hand gestures are highly correlated in most real situations. To fully exploit arm-hand correlation as well as inter-frame information, we carefully design a Spatial-Temporal Parallel Arm-Hand Motion Transformer (PAHMT) to predict the arm and hand dynamics simultaneously. We also introduce new losses to encourage the estimations to be smooth and accurate. Besides, we collect a motion capture dataset including 200K frames of hand gestures and use this data to train our model. By integrating a 2D hand pose estimation model and a 3D human pose estimation model, the proposed method can produce plausible arm and hand dynamics from monocular video. Extensive evaluations demonstrate that the proposed method has advantages over previous state-of-the-art approaches and shows robustness under various challenging scenarios.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019Searching for MobileNetV3cited by 1,984
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  7. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  8. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  9. 2017End-to-end Recovery of Human Shape and Posecited by 883
  10. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  11. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  12. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
27 more
  1. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  2. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  3. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  4. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  5. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  6. 2018Learning 3D Human Dynamics from Videocited by 315
  7. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  8. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  9. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  10. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  11. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  12. 2019Semantic Graph Convolutional Networks for 3D Human Pose Regressioncited by 204
  13. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  14. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  15. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  16. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  17. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  18. 2021PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimationcited by 88
  19. 2018Pose Flow: Efficient Online Pose Trackingcited by 80
  20. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  21. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  22. 20203D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Trainingcited by 54
  23. 2020Monocular Real-time Full Body Capture with Inter-part Correlationscited by 53
  24. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34
  25. 2021Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Camerascited by 32
  26. 2021Improving Robustness and Accuracy via Relative Information Encoding in 3D Human Pose Estimationcited by 18
  27. 2020Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videoscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2023Recovering 3D Hand Mesh Sequence from a Single Blurry Image: A New Dataset and Temporal Unfoldingcited by 10
  2. 2024Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modelingcited by 6
  3. 2023Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videoscited by 5
  4. 2023CLIP-Hand3D: Exploiting 3D Hand Pose Estimation via Context-Aware Promptingcited by 5
  5. 2023Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstructioncited by 4
  6. 2024EA-RAS: Towards Efficient and Accurate End-to-End Reconstruction of Anatomical Skeletoncited by 0
  7. 2025TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interactioncited by 0
  8. 2026REACH: Hand Pose Estimation from Room Cornerscited by 0