mlsanityData to 14 Oct 2026

MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild

arXiv 2412.13393 ↗First posted 18 Dec 2024cs.CV cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MaskHand, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MaskHand consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequence, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MaskHand achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: https://m-usamasaleem.github.io/publication/MaskHand/MaskHand.html.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

56 of 79 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Squeeze-and-Excitation Networkscited by 4,789
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
38 more
  1. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  2. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  3. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  4. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  5. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  6. 2023Reconstructing Hands in 3D with Transformerscited by 306
  7. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  8. 2021Mesh Graphormercited by 265
  9. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  10. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  11. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  12. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  13. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 200
  14. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  15. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  16. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  17. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  18. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  19. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  20. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  21. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  22. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 124
  23. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  24. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  25. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  26. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  27. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 96
  28. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  29. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  30. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  31. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  32. 2020Hierarchical Quantized Autoencoderscited by 57
  33. 2022Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Videocited by 37
  34. 2019Single Image 3D Hand Reconstruction with Mesh Convolutionscited by 32
  35. 2023A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Imagecited by 32
  36. 2016Fast and Robust Hand Tracking Using Detection-Guided Optimizationcited by 30
  37. 2024A Simple Baseline for Efficient Hand Mesh Reconstructioncited by 27
  38. 2023Sampling is Matter: Point-guided 3D Human Mesh Reconstructioncited by 24

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026SAM 3D Body: Robust Full-Body Human Mesh Recoverycited by 69
  2. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  3. 2026GeoHand: Unlocking Prior Geometry Knowledge for Monocular 3D Hand Reconstructioncited by 2
  4. 2026HandFlow: Fully Generative 4D Hand Recovery with Flow Matchingcited by 2
  5. 2026ScaleHP: Scale-Mediated Optimization of Coupled Errors for Metric-Space Hand Pose Estimationcited by 0