mlsanityData to 7 Oct 2026

Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction

arXiv 2109.01723 ↗First posted 3 Sep 2021cs.CVcited by 44

Not on the map. Why?

Abstract (arXiv)

3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR). However, this requires not only real-time speed and accurate hand pose and shape but also plausible mesh-image alignment. While existing works already achieve promising results, meeting all three requirements is very challenging. This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages: a joint stage to predict hand joints and segmentation; a mesh stage to predict a rough hand mesh; and a refine stage to fine-tune it with an offset mesh for mesh-image alignment. With careful design in the network structure and in the loss functions, we can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions. Extensive quantitative and qualitative results on benchmark datasets demonstrate that the quality of our results outperforms the state-of-the-art methods on hand-mesh/pose precision and hand-image alignment. In the end, we also showcase several real-time AR scenarios.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 3 Sep 2021.

Builds on

46 of 64 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  7. 2019PointRend: Image Segmentation as Renderingcited by 439
  8. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  9. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  10. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  11. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  12. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
34 more
  1. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  2. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  3. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  4. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  5. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  6. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  7. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  8. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  9. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  10. 2020Epipolar Transformerscited by 117
  11. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  12. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  13. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  14. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  15. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  16. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  17. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  18. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  19. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  20. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  21. 2020DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modelingcited by 49
  22. 2018Disentangling Latent Hands for Image Synthesis and Pose Estimationcited by 44
  23. 2017Using a single RGB frame for real time 3D hand pose estimation in the wildcited by 39
  24. 2020Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledgecited by 35
  25. 2020SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimationcited by 34
  26. 2020HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Mapcited by 23
  27. 2020BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networkscited by 23
  28. 2021RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Videocited by 23
  29. 2018Point-to-Pose Voting based Hand Pose Estimation using Residual Permutation Equivariant Layercited by 21
  30. 2020JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Imagecited by 20
  31. 2020AWR: Adaptive Weighting Regression for 3D Hand Pose Estimationcited by 17
  32. 2019Dual Grid Net: hand mesh vertex regression from single depth mapscited by 10
  33. 2018DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depthcited by 6
  34. 2020The Phong Surface: Efficient 3D Model Fitting using Lifted Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 16; 2024: 8; 2025: 8; 2026: 2.

44 papers.

Sort:

By area:

  1. 2023Reconstructing Hands in 3D with Transformerscited by 298
  2. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  3. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
  4. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  5. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  6. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  7. 2022LISA: Learning Implicit Shape and Appearance of Handscited by 63
  8. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  9. 2023ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstructioncited by 37
  10. 2023Neural Congealing: Aligning Images to a Joint Semantic Atlascited by 30
  11. 2022HARP: Personalized Hand Reconstruction from a Monocular RGB Videocited by 28
  12. 2024A Simple Baseline for Efficient Hand Mesh Reconstructioncited by 27
32 more
  1. 2022Interacting Hand-Object Pose Estimation via Dense Mutual Attentioncited by 22
  2. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 19
  3. 2023High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decompositioncited by 18
  4. 2023Reconstructing Interacting Hands with Interaction Prior from Monocular Imagescited by 18
  5. 2024HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Modelscited by 14
  6. 2022MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstructioncited by 11
  7. 2024HandOS: 3D Hand Reconstruction in One Stagecited by 9
  8. 2023Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Imagecited by 8
  9. 2023MeMaHand: Exploiting Mesh-Mano Interaction for Single Image Two-Hand Reconstructioncited by 8
  10. 2022TriHorn-Net: A Model for Accurate Depth-Based 3D Hand Pose Estimationcited by 7
  11. 2023HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learningcited by 7
  12. 2023DiffHand: End-to-End Hand Mesh Reconstruction via Diffusion Modelscited by 7
  13. 2024HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioningcited by 7
  14. 2026WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videoscited by 7
  15. 2022LWA-HAND: Lightweight Attention Hand for Interacting Hand Reconstructioncited by 5
  16. 2023PHRIT: Parametric Hand Representation with Implicit Templatecited by 5
  17. 2023CLIP-Hand3D: Exploiting 3D Hand Pose Estimation via Context-Aware Promptingcited by 5
  18. 2023RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touchcited by 2
  19. 2025Reconsider the Template Mesh in Deep Learning-based Mesh Reconstructioncited by 1
  20. 2025Rodrigues Network for Learning Robot Actionscited by 1
  21. 2022Pixel-Aligned Non-parametric Hand Mesh Reconstructioncited by 0
  22. 2023Lightweight Estimation of Hand Mesh and Biomechanically Feasible Kinematic Parameterscited by 0
  23. 2023HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimationcited by 0
  24. 2023Monocular 3D Hand Mesh Recovery via Dual Noise Estimationcited by 0
  25. 2024HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB imagescited by 0
  26. 2024STMR: Spiral Transformer for Hand Mesh Reconstructioncited by 0
  27. 2025Direction-Aware Hybrid Representation Learning for 3D Hand Pose and Shape Estimationcited by 0
  28. 2025M3DHMR: Monocular 3D Hand Mesh Recoverycited by 0
  29. 2025EHPE: A Segmented Architecture for Enhanced Hand Pose Estimationcited by 0
  30. 2025Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priorscited by 0
  31. 2025Cascaded Diffusion Framework for Probabilistic Coarse-to-Fine Hand Pose Estimationcited by 0
  32. 2026MASS: Mesh-inellipse Aligned Deformable Surfel Splatting for Hand Reconstruction and Rendering from Egocentric Monocular Videocited by 0