# UV-Based 3D Hand-Object Reconstruction with Grasp Optimization

arXiv: https://arxiv.org/abs/2211.13429 · HTML: https://mlsanity.com/paper/2211.13429/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 70 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [Hand-Object Contact Consistency Reasoning for Human Grasps Generation](https://mlsanity.com/paper/2104.03304/) (2021)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction](https://mlsanity.com/paper/2004.13449/) (2020)
- [CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction](https://mlsanity.com/paper/2012.00924/) (2020)

## Built on by (most cited first, 8 of 8)

- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model](https://mlsanity.com/paper/2407.12727/) (2024)
- [MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild](https://mlsanity.com/paper/2412.13393/) (2024)
- [Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction](https://mlsanity.com/paper/2305.00646/) (2023)
- [Nonrigid Object Contact Estimation With Regional Unwrapping Transformer](https://mlsanity.com/paper/2308.14074/) (2023)
- [NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interaction](https://mlsanity.com/paper/2402.05532/) (2024)
- [A Versatile and Differentiable Hand-Object Interaction Representation](https://mlsanity.com/paper/2409.16855/) (2024)
- [TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interaction](https://mlsanity.com/paper/2506.00953/) (2025)

## Abstract (arXiv)

We propose a novel framework for 3D hand shape reconstruction and hand-object grasp optimization from a single RGB image. The representation of hand-object contact regions is critical for accurate reconstructions. Instead of approximating the contact regions with sparse points, as in previous works, we propose a dense representation in the form of a UV coordinate map. Furthermore, we introduce inference-time optimization to fine-tune the grasp and improve interactions between the hand and the object. Our pipeline increases hand shape reconstruction accuracy and produces a vibrant hand texture. Experiments on datasets such as Ho3D, FreiHAND, and DexYCB reveal that our proposed method outperforms the state-of-the-art.