# UST-Hand: An Uncertainty-aware Spatiotemporal Point Cloud Interaction Network for 3D Self-supervised Hand Pose Estimation

arXiv: https://arxiv.org/abs/2605.17742 · HTML: https://mlsanity.com/paper/2605.17742/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 70 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Probabilistic Modeling for Human Mesh Recovery](https://mlsanity.com/paper/2108.11944/) (2021)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping](https://mlsanity.com/paper/2112.11454/) (2021)
- [Human Pose Regression with Residual Log-likelihood Estimation](https://mlsanity.com/paper/2107.11291/) (2021)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020)
- [D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions](https://mlsanity.com/paper/2112.03028/) (2021)
- [Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints](https://mlsanity.com/paper/2003.09282/) (2020)
- [Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network](https://mlsanity.com/paper/1904.05547/) (2019)
- [Monocular 3D Human Pose Estimation by Generation and Ordinal Ranking](https://mlsanity.com/paper/1904.01324/) (2019)
- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data](https://mlsanity.com/paper/2011.00980/) (2020)
- [Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild](https://mlsanity.com/paper/2110.00990/) (2021)
- [Contrastive Representation Learning for Hand Shape Estimation](https://mlsanity.com/paper/2106.04324/) (2021)
- [GraphMDN: Leveraging graph structure and deep learning to solve inverse problems](https://mlsanity.com/paper/2010.13668/) (2020)
- [HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset](https://mlsanity.com/paper/2107.00887/) (2021)

## Abstract (arXiv)

Manually annotating accurate 3D hand poses is extremely time-consuming and labor-intensive. Existing self-supervised hand pose estimation methods leverage the discrepancy between input images and rendered outputs, or multi-view consistency constraints, as the driving force to optimize networks and progressively refine pose accuracy. However, these methods are highly susceptible to noisy pseudo-labels and overlook the importance of fully exploiting fine-grained spatial correlations, which undermines the stability of model training. To address these issues, we propose UST-Hand, a self-supervised learning framework that estimates uncertainty distribution of hand pose and constructs a probabilistic point cloud feature space, which enables the complex spatiotemporal relationship modeling. UST-Hand employs a conditional normalizing flow model to capture hand pose distributions and samples diverse hypotheses, facilitating robust learning under noisy pseudo-labels supervision with enhanced stability. These multi-hypothesis are mapped to a unified probabilistic 3D point cloud space for multi-view and temporal feature interaction, comprehensively exploring hand motion patterns and fine-grained spatial correlations. Extensive experiments on three challenging datasets demonstrate that UST-Hand achieves state-of-the-art performance, outperforming existing self-supervised methods by up to 37.8% in Mean Per Vertex Position Error (MPVPE).