# A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Image

arXiv: https://arxiv.org/abs/2304.14299 · HTML: https://mlsanity.com/paper/2304.14299/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation](https://mlsanity.com/paper/2104.14639/) (2021)
- [Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints](https://mlsanity.com/paper/2003.09282/) (2020)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis](https://mlsanity.com/paper/2109.05488/) (2021)
- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling](https://mlsanity.com/paper/2102.03725/) (2021)
- [HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset](https://mlsanity.com/paper/2107.00887/) (2021)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [A Multi-modal Machine Learning Approach and Toolkit to Automate Recognition of Early Stages of Dementia among British Sign Language Users](https://mlsanity.com/paper/2010.00536/) (2020)

## Built on by (most cited first, 30 of 32)

- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [3D Hand Pose Estimation in Everyday Egocentric Images](https://mlsanity.com/paper/2312.06583/) (2023)
- [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025)
- [HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions](https://mlsanity.com/paper/2403.18575/) (2024)
- [3D Points Splatting for Real-Time Dynamic Hand Reconstruction](https://mlsanity.com/paper/2312.13770/) (2023)
- [HandOS: 3D Hand Reconstruction in One Stage](https://mlsanity.com/paper/2412.01537/) (2024)
- [BiTT: Bi-directional Texture Reconstruction of Interacting Two Hands from a Single Image](https://mlsanity.com/paper/2403.08262/) (2024)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [Multi-view Hand Reconstruction with a Point-Embedded Transformer](https://mlsanity.com/paper/2408.10581/) (2024)
- [Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidance](https://mlsanity.com/paper/2407.12307/) (2024)
- [MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild](https://mlsanity.com/paper/2412.13393/) (2024)
- [Diffusion-based 3D Hand Motion Recovery with Intuitive Physics](https://mlsanity.com/paper/2508.01835/) (2025)
- [PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery](https://mlsanity.com/paper/2603.26068/) (2026)
- [Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars](https://mlsanity.com/paper/2410.08840/) (2024)
- [SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit 3D Meshes](https://mlsanity.com/paper/2509.21859/) (2025)
- [EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning](https://mlsanity.com/paper/2601.19850/) (2026)
- [3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario](https://mlsanity.com/paper/2403.01733/) (2024)
- [Rodrigues Network for Learning Robot Actions](https://mlsanity.com/paper/2506.02618/) (2025)
- [AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation](https://mlsanity.com/paper/2603.25726/) (2026)
- [ADL4D: Towards A Contextually Rich Dataset for 4D Activities of Daily Living](https://mlsanity.com/paper/2402.17758/) (2024)
- [HandSCS: Structural Coordinate Space for Animatable Hand Gaussian Splatting](https://mlsanity.com/paper/2503.14736/) (2025)
- [From 2D Alignment to 3D Plausibility: Unifying Heterogeneous 2D Priors and Penetration-Free Diffusion for Occlusion-Robust Two-Hand Reconstruction](https://mlsanity.com/paper/2503.17788/) (2025)
- [Direction-Aware Hybrid Representation Learning for 3D Hand Pose and Shape Estimation](https://mlsanity.com/paper/2504.01298/) (2025)
- [Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priors](https://mlsanity.com/paper/2508.09629/) (2025)
- [Learning Correlation-aware Aleatoric Uncertainty for 3D Hand Pose Estimation](https://mlsanity.com/paper/2509.01242/) (2025)
- [Cascaded Diffusion Framework for Probabilistic Coarse-to-Fine Hand Pose Estimation](https://mlsanity.com/paper/2510.00527/) (2025)
- [MASS: Mesh-inellipse Aligned Deformable Surfel Splatting for Hand Reconstruction and Rendering from Egocentric Monocular Video](https://mlsanity.com/paper/2604.08943/) (2026)
- [Embodied Operators and Benchmarking: Toward Reusable and Deployable Embodied Intelligence Systems](https://mlsanity.com/paper/2607.03283/) (2026)

## Abstract (arXiv)

Recently, deep learning based approaches have shown promising results in 3D hand reconstruction from a single RGB image. These approaches can be roughly divided into model-based approaches, which are heavily dependent on the model's parameter space, and model-free approaches, which require large numbers of 3D ground truths to reduce depth ambiguity and struggle in weakly-supervised scenarios. To overcome these issues, we propose a novel probabilistic model to achieve the robustness of model-based approaches and reduced dependence on the model's parameter space of model-free approaches. The proposed probabilistic model incorporates a model-based network as a prior-net to estimate the prior probability distribution of joints and vertices. An Attention-based Mesh Vertices Uncertainty Regression (AMVUR) model is proposed to capture dependencies among vertices and the correlation between joints and mesh vertices to improve their feature representation. We further propose a learning based occlusion-aware Hand Texture Regression model to achieve high-fidelity texture reconstruction. We demonstrate the flexibility of the proposed probabilistic model to be trained in both supervised and weakly-supervised scenarios. The experimental results demonstrate our probabilistic model's state-of-the-art accuracy in 3D hand and texture reconstruction from a single image in both training schemes, including in the presence of severe occlusions.