# Contrastive Representation Learning for Hand Shape Estimation

arXiv: https://arxiv.org/abs/2106.04324 · HTML: https://mlsanity.com/paper/2106.04324/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)
- [Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints](https://mlsanity.com/paper/2003.09282/) (2020)
- [Cross-modal Deep Variational Hand Pose Estimation](https://mlsanity.com/paper/1803.11404/) (2018)
- [Learning Monocular 3D Human Pose Estimation from Multi-view Images](https://mlsanity.com/paper/1803.04775/) (2018)
- [DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling](https://mlsanity.com/paper/2008.08213/) (2020)

## Built on by (most cited first, 30 of 34)

- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [Interacting Hand-Object Pose Estimation via Dense Mutual Attention](https://mlsanity.com/paper/2211.08805/) (2022)
- [SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale](https://mlsanity.com/paper/2406.06907/) (2024)
- [OHTA: One-shot Hand Avatar via Data-driven Implicit Priors](https://mlsanity.com/paper/2402.18969/) (2024)
- [Efficient Annotation and Learning for 3D Hand Pose Estimation: A Survey](https://mlsanity.com/paper/2206.02257/) (2022)
- [TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning](https://mlsanity.com/paper/2209.00489/) (2022)
- [Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics](https://mlsanity.com/paper/2409.04033/) (2024)
- [Domain Adaptive Hand Keypoint and Pixel Localization in the Wild](https://mlsanity.com/paper/2203.08344/) (2022)
- [HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image](https://mlsanity.com/paper/2309.07891/) (2023)
- [HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning](https://mlsanity.com/paper/2302.00988/) (2023)
- [Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2303.05370/) (2023)
- [Towards Robust and Expressive Whole-body Human Pose and Shape Estimation](https://mlsanity.com/paper/2312.08730/) (2023)
- [Self-Supervised Learning with Generative Adversarial Networks for Electron Microscopy](https://mlsanity.com/paper/2402.18286/) (2024)
- [JGHand: Joint-Driven Animatable Hand Avater via 3D Gaussian Splatting](https://mlsanity.com/paper/2501.19088/) (2025)
- [EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation](https://mlsanity.com/paper/2605.05712/) (2026)
- [EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera](https://mlsanity.com/paper/2605.12498/) (2026)
- [Multi-view Image-based Hand Geometry Refinement using Differentiable Monte Carlo Ray Tracing](https://mlsanity.com/paper/2107.05509/) (2021)
- [Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video](https://mlsanity.com/paper/2202.08418/) (2022)
- [Self-Supervised Contrastive Representation Learning for 3D Mesh Segmentation](https://mlsanity.com/paper/2208.04278/) (2022)
- [SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation](https://mlsanity.com/paper/2402.01389/) (2024)
- [HandOcc: NeRF-based Hand Rendering with Occupancy Networks](https://mlsanity.com/paper/2505.02079/) (2025)
- [GeoHand: Unlocking Prior Geometry Knowledge for Monocular 3D Hand Reconstruction](https://mlsanity.com/paper/2605.17354/) (2026)
- [MBW: Multi-view Bootstrapping in the Wild](https://mlsanity.com/paper/2210.01721/) (2022)
- [ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty](https://mlsanity.com/paper/2303.10042/) (2023)
- [Self-supervised Optimization of Hand Pose Estimation using Anatomical Features and Iterative Learning](https://mlsanity.com/paper/2307.03007/) (2023)
- [HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimation](https://mlsanity.com/paper/2307.16061/) (2023)
- [Automated Patient Positioning with Learned 3D Hand Gestures](https://mlsanity.com/paper/2407.14903/) (2024)
- [Direction-Aware Hybrid Representation Learning for 3D Hand Pose and Shape Estimation](https://mlsanity.com/paper/2504.01298/) (2025)
- [Fitting Image Diffusion Models on Video Datasets](https://mlsanity.com/paper/2509.03794/) (2025)

## Abstract (arXiv)

This work presents improvements in monocular hand shape estimation by building on top of recent advances in unsupervised learning. We extend momentum contrastive learning and contribute a structured collection of hand images, well suited for visual representation learning, which we call HanCo. We find that the representation learned by established contrastive learning methods can be improved significantly by exploiting advanced background removal techniques and multi-view information. These allow us to generate more diverse instance pairs than those obtained by augmentations commonly used in exemplar based approaches. Our method leads to a more suitable representation for the hand shape estimation task and shows a 4.7% reduction in mesh error and a 3.6% improvement in F-score compared to an ImageNet pretrained baseline. We make our benchmark dataset publicly available, to encourage further research into this direction.