# OmniHands: Towards Robust 4D Hand Mesh Recovery via A Versatile Transformer

arXiv: https://arxiv.org/abs/2405.20330 · HTML: https://mlsanity.com/paper/2405.20330/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2024.

## Built on by (most cited first, 7 of 7)

- [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025)
- [The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction](https://mlsanity.com/paper/2606.30308/) (2026)
- [ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery](https://mlsanity.com/paper/2608.20308/) (2026)
- [MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision](https://mlsanity.com/paper/2609.04958/) (2026)
- [WildGHand: Learning Anti-Perturbation Gaussian Hand Avatars from Monocular In-the-Wild Videos](https://mlsanity.com/paper/2602.20556/) (2026)
- [InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video](https://mlsanity.com/paper/2609.35743/) (2026)
- [EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations](https://mlsanity.com/paper/2610.16256/) (2026)

## Abstract (arXiv)

In this paper, we introduce OmniHands, a universal approach to recovering interactive hand meshes and their relative movement from monocular or multi-view inputs. Our approach addresses two major limitations of previous methods: lacking a unified solution for handling various hand image inputs and neglecting the positional relationship of two hands within images. To overcome these challenges, we develop a universal architecture with novel tokenization and contextual feature fusion strategies, capable of adapting to a variety of tasks. Specifically, we propose a Relation-aware Two-Hand Tokenization (RAT) method to embed positional relation information into the hand tokens. In this way, our network can handle both single-hand and two-hand inputs and explicitly leverage relative hand positions, facilitating the reconstruction of intricate hand interactions in real-world scenarios. As such tokenization indicates the relative relationship of two hands, it also supports more effective feature fusion. To this end, we further develop a 4D Interaction Reasoning (FIR) module to fuse hand tokens in 4D with attention and decode them into 3D hand meshes and relative temporal movements. The efficacy of our approach is validated on several benchmark datasets. The results on in-the-wild videos and real-world scenarios demonstrate the superior performances of our approach for interactive hand reconstruction. More video results can be found on the project page: https://OmniHand.github.io.