# Multi-View Video-Based 3D Hand Pose Estimation arXiv: https://arxiv.org/abs/2109.11747 ยท HTML: https://mlsanity.com/paper/2109.11747/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 62 references that are in the map. - First posted on arXiv 24 Sep 2021. ## Builds on (most cited first, 30 of 34) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019) - [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019) - [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019) - [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020) - [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017) - [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017) - [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017) - [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019) - [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019) - [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017) - [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020) - [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020) - [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019) - [HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation](https://mlsanity.com/paper/2004.00060/) (2020) - [Hand Pose Estimation via Latent 2.5D Heatmap Regression](https://mlsanity.com/paper/1804.09534/) (2018) - [Cross-modal Deep Variational Hand Pose Estimation](https://mlsanity.com/paper/1803.11404/) (2018) - [Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor](https://mlsanity.com/paper/1704.02201/) (2017) - [BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis](https://mlsanity.com/paper/1704.02612/) (2017) - [An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2007.12392/) (2020) - [Using a single RGB frame for real time 3D hand pose estimation in the wild](https://mlsanity.com/paper/1712.03866/) (2017) - [3D Hand Pose Tracking and Estimation Using Stereo Matching](https://mlsanity.com/paper/1610.07214/) (2016) - [SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation](https://mlsanity.com/paper/2007.05168/) (2020) - [Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics](https://mlsanity.com/paper/2007.12287/) (2020) - [Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction](https://mlsanity.com/paper/2003.13764/) (2020) - [Depth-based hand pose estimation: methods, data, and challenges](https://mlsanity.com/paper/1504.06378/) (2015) - [MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation](https://mlsanity.com/paper/2012.03206/) (2020) ## Built on by (most cited first, 6 of 6) - [Interacting Hand-Object Pose Estimation via Dense Mutual Attention](https://mlsanity.com/paper/2211.08805/) (2022) - [Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation](https://mlsanity.com/paper/2303.04991/) (2023) - [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025) - [RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch](https://mlsanity.com/paper/2302.14365/) (2023) - [Learning Sequential Contexts using Transformer for 3D Hand Pose Estimation](https://mlsanity.com/paper/2206.00171/) (2022) - [A multimodal gesture recognition dataset for desktop human-computer interaction](https://mlsanity.com/paper/2401.03828/) (2024) ## Abstract (arXiv) Hand pose estimation (HPE) can be used for a variety of human-computer interaction applications such as gesture-based control for physical or virtual/augmented reality devices. Recent works have shown that videos or multi-view images carry rich information regarding the hand, allowing for the development of more robust HPE systems. In this paper, we present the Multi-View Video-Based 3D Hand (MuViHand) dataset, consisting of multi-view videos of the hand along with ground-truth 3D pose labels. Our dataset includes more than 402,000 synthetic hand images available in 4,560 videos. The videos have been simultaneously captured from six different angles with complex backgrounds and random levels of dynamic lighting. The data has been captured from 10 distinct animated subjects using 12 cameras in a semi-circle topology where six tracking cameras only focus on the hand and the other six fixed cameras capture the entire body. Next, we implement MuViHandNet, a neural pipeline consisting of image encoders for obtaining visual embeddings of the hand, recurrent learners to learn both temporal and angular sequential information, and graph networks with U-Net architectures to estimate the final 3D pose information. We perform extensive experiments and show the challenging nature of this new dataset as well as the effectiveness of our proposed method. Ablation studies show the added value of each component in MuViHandNet, as well as the benefit of having temporal and sequential information in the dataset.