mlsanityData to 7 Oct 2026

Multi-View Video-Based 3D Hand Pose Estimation

arXiv 2109.11747 ↗First posted 24 Sep 2021cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Hand pose estimation (HPE) can be used for a variety of human-computer interaction applications such as gesture-based control for physical or virtual/augmented reality devices. Recent works have shown that videos or multi-view images carry rich information regarding the hand, allowing for the development of more robust HPE systems. In this paper, we present the Multi-View Video-Based 3D Hand (MuViHand) dataset, consisting of multi-view videos of the hand along with ground-truth 3D pose labels. Our dataset includes more than 402,000 synthetic hand images available in 4,560 videos. The videos have been simultaneously captured from six different angles with complex backgrounds and random levels of dynamic lighting. The data has been captured from 10 distinct animated subjects using 12 cameras in a semi-circle topology where six tracking cameras only focus on the hand and the other six fixed cameras capture the entire body. Next, we implement MuViHandNet, a neural pipeline consisting of image encoders for obtaining visual embeddings of the hand, recurrent learners to learn both temporal and angular sequential information, and graph networks with U-Net architectures to estimate the final 3D pose information. We perform extensive experiments and show the challenging nature of this new dataset as well as the effectiveness of our proposed method. Ablation studies show the added value of each component in MuViHandNet, as well as the benefit of having temporal and sequential information in the dataset.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

34 of 62 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019Graph U-Netscited by 489
  6. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  7. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  8. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  9. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  10. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  11. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  12. 20193D Hand Shape and Pose from Images in the Wildcited by 190
22 more
  1. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  2. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  3. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  4. 2020Epipolar Transformerscited by 117
  5. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  6. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  7. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  8. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  9. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  10. 2017BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysiscited by 46
  11. 2020An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Cloudscited by 45
  12. 2017Using a single RGB frame for real time 3D hand pose estimation in the wildcited by 39
  13. 20163D Hand Pose Tracking and Estimation Using Stereo Matchingcited by 35
  14. 2020SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimationcited by 34
  15. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34
  16. 2020Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interactioncited by 24
  17. 2015Depth-based hand pose estimation: methods, data, and challengescited by 16
  18. 2020MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimationcited by 15
  19. 2017Occlusion-aware Hand Pose Estimation Using Hierarchical Mixture Density Networkcited by 11
  20. 2019Hand Pose Estimation: A Surveycited by 8
  21. 2018HandyNet: A One-stop Solution to Detect, Segment, Localize & Analyze Driver Handscited by 6
  22. 2021Multi-Perspective LSTM for Joint Visual Representation Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 1.

6 papers.

  1. 2022Interacting Hand-Object Pose Estimation via Dense Mutual Attentioncited by 22
  2. 2023Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimationcited by 21
  3. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 19
  4. 2023RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touchcited by 2
  5. 2022Learning Sequential Contexts using Transformer for 3D Hand Pose Estimationcited by 1
  6. 2024A multimodal gesture recognition dataset for desktop human-computer interactioncited by 0