mlsanityData to 7 Oct 2026

UV-Based 3D Hand-Object Reconstruction with Grasp Optimization

arXiv 2211.13429 ↗First posted 24 Nov 2022cs.CV cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a novel framework for 3D hand shape reconstruction and hand-object grasp optimization from a single RGB image. The representation of hand-object contact regions is critical for accurate reconstructions. Instead of approximating the contact regions with sparse points, as in previous works, we propose a dense representation in the form of a UV coordinate map. Furthermore, we introduce inference-time optimization to fine-tune the grasp and improve interactions between the hand and the object. Our pipeline increases hand shape reconstruction accuracy and produces a vibrant hand texture. Experiments on datasets such as Ho3D, FreiHAND, and DexYCB reveal that our proposed method outperforms the state-of-the-art.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 50 of its 70 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

50 of 70 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  5. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  8. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  9. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  10. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  11. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  12. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
38 more
  1. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  2. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  3. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  4. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  5. 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 267
  6. 2021Mesh Graphormercited by 264
  7. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  8. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  9. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  10. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  11. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  12. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  13. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  14. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  15. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  16. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  17. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  18. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  19. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  20. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  21. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  22. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  23. 2019Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformationcited by 71
  24. 20203D Human Mesh Regression with Dense Correspondencecited by 66
  25. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  26. 2017UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognitioncited by 62
  27. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  28. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  29. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  30. 2021A Skeleton-Driven Neural Occupancy Representation for Articulated Handscited by 50
  31. 2018Disentangling Latent Hands for Image Synthesis and Pose Estimationcited by 44
  32. 2018Exploring Spatial Context for 3D Semantic Segmentation of Point Cloudscited by 33
  33. 2021HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Datasetcited by 25
  34. 2019DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Imagecited by 20
  35. 20173DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Cloudscited by 12
  36. 2017Dynamics Based 3D Skeletal Hand Trackingcited by 10
  37. 2019Dual Grid Net: hand mesh vertex regression from single depth mapscited by 10
  38. 2020StyleUV: Diverse and High-fidelity UV Map Generative Modelcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 1.

8 papers.

  1. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  2. 2024NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Modelcited by 10
  3. 2024MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wildcited by 5
  4. 2023Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstructioncited by 4
  5. 2023Nonrigid Object Contact Estimation With Regional Unwrapping Transformercited by 3
  6. 2024NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interactioncited by 1
  7. 2024A Versatile and Differentiable Hand-Object Interaction Representationcited by 1
  8. 2025TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interactioncited by 0