# A Skeleton-Driven Neural Occupancy Representation for Articulated Hands

arXiv: https://arxiv.org/abs/2109.11399 · HTML: https://mlsanity.com/paper/2109.11399/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 71 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 30 of 57)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019)

## Built on by (most cited first, 30 of 50)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions](https://mlsanity.com/paper/2112.03028/) (2021)
- [TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding](https://mlsanity.com/paper/2401.08399/) (2024)
- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022)
- [ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation](https://mlsanity.com/paper/2309.03891/) (2023)
- [ContactGen: Generative Contact Modeling for Grasp Generation](https://mlsanity.com/paper/2310.03740/) (2023)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video](https://mlsanity.com/paper/2211.12782/) (2022)
- [UGG: Unified Generative Grasping](https://mlsanity.com/paper/2311.16917/) (2023)
- [SemGrasp: Semantic Grasp Generation via Language Aligned Discretization](https://mlsanity.com/paper/2404.03590/) (2024)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion](https://mlsanity.com/paper/2403.17422/) (2024)
- [GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency](https://mlsanity.com/paper/2308.11617/) (2023)
- [Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes](https://mlsanity.com/paper/2302.14348/) (2023)
- [RelightableHands: Efficient Neural Relighting of Articulated Hand Models](https://mlsanity.com/paper/2302.04866/) (2023)
- [ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion](https://mlsanity.com/paper/2409.09300/) (2024)
- [GEARS: Local Geometry-aware Hand-object Interaction Synthesis](https://mlsanity.com/paper/2404.01758/) (2024)
- [OHTA: One-shot Hand Avatar via Data-driven Implicit Priors](https://mlsanity.com/paper/2402.18969/) (2024)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views](https://mlsanity.com/paper/2308.11198/) (2023)
- [Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics](https://mlsanity.com/paper/2409.04033/) (2024)
- [CHORD: Category-level Hand-held Object Reconstruction via Shape Deformation](https://mlsanity.com/paper/2308.10574/) (2023)
- [FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow](https://mlsanity.com/paper/2307.08100/) (2023)
- [Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images](https://mlsanity.com/paper/2308.11015/) (2023)
- [Stability-driven Contact Reconstruction From Monocular Color Images](https://mlsanity.com/paper/2205.00848/) (2022)
- [UV-Based 3D Hand-Object Reconstruction with Grasp Optimization](https://mlsanity.com/paper/2211.13429/) (2022)
- [BiTT: Bi-directional Texture Reconstruction of Interacting Two Hands from a Single Image](https://mlsanity.com/paper/2403.08262/) (2024)
- [Human 3D Avatar Modeling with Implicit Neural Representation: A Brief Survey](https://mlsanity.com/paper/2306.03576/) (2023)

## Abstract (arXiv)

We present Hand ArticuLated Occupancy (HALO), a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces and can be used in end-to-end trainable architectures. Unlike existing statistical parametric hand models (e.g.~MANO), HALO directly leverages 3D joint skeleton as input and produces a neural occupancy volume representing the posed hand surface. The key benefits of HALO are (1) it is driven by 3D key points, which have benefits in terms of accuracy and are easier to learn for neural networks than the latent hand-model parameters; (2) it provides a differentiable volumetric occupancy representation of the posed hand; (3) it can be trained end-to-end, allowing the formulation of losses on the hand surface that benefit the learning of 3D keypoints. We demonstrate the applicability of HALO to the task of conditional generation of hands that grasp 3D objects. The differentiable nature of HALO is shown to improve the quality of the synthesized hands both in terms of physical plausibility and user preference.