# MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints

arXiv: https://arxiv.org/abs/2404.10227 · HTML: https://mlsanity.com/paper/2404.10227/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video](https://mlsanity.com/paper/2011.08627/) (2020)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction](https://mlsanity.com/paper/2004.13449/) (2020)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction](https://mlsanity.com/paper/2012.00924/) (2020)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields](https://mlsanity.com/paper/2207.13807/) (2022)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis](https://mlsanity.com/paper/2109.05488/) (2021)
- [TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement](https://mlsanity.com/paper/2205.07982/) (2022)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation](https://mlsanity.com/paper/2007.05168/) (2020)
- [Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation](https://mlsanity.com/paper/1904.13041/) (2019)
- [BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks](https://mlsanity.com/paper/2008.05079/) (2020)
- [Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation](https://mlsanity.com/paper/2303.04991/) (2023)
- [H2O: A Benchmark for Visual Human-human Object Handover Analysis](https://mlsanity.com/paper/2104.11466/) (2021)
- [HandTailor: Towards High-Precision Monocular 3D Hand Recovery](https://mlsanity.com/paper/2102.09244/) (2021)
- [RCareWorld: A Human-centric Simulation World for Caregiving Robots](https://mlsanity.com/paper/2210.10821/) (2022)
- [RFUniverse: A Multiphysics Simulation Platform for Embodied AI](https://mlsanity.com/paper/2202.00199/) (2022)

## Built on by (most cited first, 9 of 9)

- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Diffusion-based 3D Hand Motion Recovery with Intuitive Physics](https://mlsanity.com/paper/2508.01835/) (2025)
- [PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery](https://mlsanity.com/paper/2603.26068/) (2026)
- [AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation](https://mlsanity.com/paper/2603.25726/) (2026)
- [HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB images](https://mlsanity.com/paper/2405.01066/) (2024)
- [When Dance Video Archives Challenge Computer Vision](https://mlsanity.com/paper/2505.07249/) (2025)
- [VPHO: Joint Visual-Physical Cue Learning and Aggregation for Hand-Object Pose Estimation](https://mlsanity.com/paper/2511.12030/) (2025)
- [A Multi-View Pipeline and Benchmark Dataset for 3D Hand Pose Estimation in Surgery](https://mlsanity.com/paper/2601.15918/) (2026)
- [MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild](https://mlsanity.com/paper/2609.16684/) (2026)

## Abstract (arXiv)

This work proposes a novel learning framework for visual hand dynamics analysis that takes into account the physiological aspects of hand motion. The existing models, which are simplified joint-actuated systems, often produce unnatural motions. To address this, we integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create a new model, MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We further propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the efficacy of the BioPR is conducted in two separate parts. The accuracy of MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked against two large-scale public datasets and two recent state-of-the-art methods. The results demonstrate that our approach consistently improves the baseline methods both quantitatively and qualitatively.