# Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation

arXiv: https://arxiv.org/abs/2106.05954 · HTML: https://mlsanity.com/paper/2106.05954/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [HP-GAN: Probabilistic 3D human motion prediction via GAN](https://mlsanity.com/paper/1711.09561/) (2017)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction](https://mlsanity.com/paper/2004.13449/) (2020)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)

## Built on by (most cited first, 7 of 7)

- [HMP: Hand Motion Priors for Pose and Shape Estimation from Video](https://mlsanity.com/paper/2312.16737/) (2023)
- [Efficient Annotation and Learning for 3D Hand Pose Estimation: A Survey](https://mlsanity.com/paper/2206.02257/) (2022)
- [TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning](https://mlsanity.com/paper/2209.00489/) (2022)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [S2Contact: Graph-based Network for 3D Hand-Object Contact Estimation with Semi-Supervised Learning](https://mlsanity.com/paper/2208.00874/) (2022)
- [Multi-view Image-based Hand Geometry Refinement using Differentiable Monte Carlo Ray Tracing](https://mlsanity.com/paper/2107.05509/) (2021)
- [Hand-4DGS: Feed-Forward 3D Gaussian Splatting for 4D Hand Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2606.19156/) (2026)

## Abstract (arXiv)

Hand pose estimation is difficult due to different environmental conditions, object- and self-occlusion as well as diversity in hand shape and appearance. Exhaustively covering this wide range of factors in fully annotated datasets has remained impractical, posing significant challenges for generalization of supervised methods. Embracing this challenge, we propose to combine ideas from adversarial training and motion modelling to tap into unlabeled videos. To this end we propose what to the best of our knowledge is the first motion model for hands and show that an adversarial formulation leads to better generalization properties of the hand pose estimator via semi-supervised training on unlabeled video sequences. In this setting, the pose predictor must produce a valid sequence of hand poses, as determined by a discriminative adversary. This adversary reasons both on the structural as well as temporal domain, effectively exploiting the spatio-temporal structure in the task. The main advantage of our approach is that we can make use of unpaired videos and joint sequence data both of which are much easier to attain than paired training data. We perform extensive evaluation, investigating essential components needed for the proposed framework and empirically demonstrate in two challenging settings that the proposed approach leads to significant improvements in pose estimation accuracy. In the lowest label setting, we attain an improvement of 40% in absolute mean joint error.