# HARP: Personalized Hand Reconstruction from a Monocular RGB Video

arXiv: https://arxiv.org/abs/2212.09530 · HTML: https://mlsanity.com/paper/2212.09530/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 92 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)

## Built on by (most cited first, 28 of 28)

- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [MANUS: Markerless Grasp Capture using Articulated 3D Gaussians](https://mlsanity.com/paper/2312.02137/) (2023)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision](https://mlsanity.com/paper/2409.02224/) (2024)
- [OHTA: One-shot Hand Avatar via Data-driven Implicit Priors](https://mlsanity.com/paper/2402.18969/) (2024)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [Authentic Hand Avatar from a Phone Scan via Universal Hand Model](https://mlsanity.com/paper/2405.07933/) (2024)
- [3D Points Splatting for Real-Time Dynamic Hand Reconstruction](https://mlsanity.com/paper/2312.13770/) (2023)
- [BiTT: Bi-directional Texture Reconstruction of Interacting Two Hands from a Single Image](https://mlsanity.com/paper/2403.08262/) (2024)
- [DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation](https://mlsanity.com/paper/2506.03103/) (2025)
- [Advances in Radiance Field for Dynamic Scene: From Neural Field to Gaussian Field](https://mlsanity.com/paper/2505.10049/) (2025)
- [Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering](https://mlsanity.com/paper/2407.05680/) (2024)
- [Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars](https://mlsanity.com/paper/2410.08840/) (2024)
- [SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit 3D Meshes](https://mlsanity.com/paper/2509.21859/) (2025)
- [PALM: A Dataset and Baseline for Learning Multi-subject Hand Prior](https://mlsanity.com/paper/2511.05403/) (2025)
- [Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves](https://mlsanity.com/paper/2603.20850/) (2026)
- [Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images](https://mlsanity.com/paper/2307.06038/) (2023)
- [HiFiHR: Enhancing 3D Hand Reconstruction from a Single Image via High-Fidelity Texture](https://mlsanity.com/paper/2308.13628/) (2023)
- [XHand: Real-time Expressive Hand Avatar](https://mlsanity.com/paper/2407.21002/) (2024)
- [HandOcc: NeRF-based Hand Rendering with Occupancy Networks](https://mlsanity.com/paper/2505.02079/) (2025)
- [HandSCS: Structural Coordinate Space for Animatable Hand Gaussian Splatting](https://mlsanity.com/paper/2503.14736/) (2025)
- [Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction](https://mlsanity.com/paper/2511.14540/) (2025)
- [HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance](https://mlsanity.com/paper/2604.04425/) (2026)
- [MASS: Mesh-inellipse Aligned Deformable Surfel Splatting for Hand Reconstruction and Rendering from Egocentric Monocular Video](https://mlsanity.com/paper/2604.08943/) (2026)
- [PHAF-Personalized Hand Avatars in a Flash](https://mlsanity.com/paper/2606.03420/) (2026)
- [VEPHand: View-Efficient Photometric Hand Performance Capture at Scale](https://mlsanity.com/paper/2606.15966/) (2026)
- [Hand-4DGS: Feed-Forward 3D Gaussian Splatting for 4D Hand Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2606.19156/) (2026)
- [OASIS: Occlusion-aware Single-image Hand Avatar Reconstruction via 3D Gaussian Splatting](https://mlsanity.com/paper/2607.29633/) (2026)

## Abstract (arXiv)

We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry. In contrast to the major trend of neural implicit representations, HARP models a hand with a mesh-based parametric hand model, a vertex displacement map, a normal map, and an albedo without any neural components. As validated by our experiments, the explicit nature of our representation enables a truly scalable, robust, and efficient approach to hand avatar creation. HARP is optimized via gradient descent from a short sequence captured by a hand-held mobile phone and can be directly used in AR/VR applications with real-time rendering capability. To enable this, we carefully design and implement a shadow-aware differentiable rendering scheme that is robust to high degree articulations and self-shadowing regularly present in hand motion sequences, as well as challenging lighting conditions. It also generalizes to unseen poses and novel viewpoints, producing photo-realistic renderings of hand animations performing highly-articulated motions. Furthermore, the learned HARP representation can be used for improving 3D hand pose estimation quality in challenging viewpoints. The key advantages of HARP are validated by the in-depth analyses on appearance reconstruction, novel-view and novel pose synthesis, and 3D hand pose refinement. It is an AR/VR-ready personalized hand representation that shows superior fidelity and scalability.