# Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models

arXiv: https://arxiv.org/abs/2605.09258 · HTML: https://mlsanity.com/paper/2605.09258/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 21 of 21)

- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [Equinox: neural networks in JAX via callable PyTrees and filtered transformations](https://mlsanity.com/paper/2111.00254/) (2021)
- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [SAM 3D Body: Robust Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2602.15989/) (2026)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats](https://mlsanity.com/paper/2212.14474/) (2022)
- [LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion](https://mlsanity.com/paper/2311.02496/) (2023)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [Reconstructing Humans with a Biomechanically Accurate Skeleton](https://mlsanity.com/paper/2503.21751/) (2025)
- [Differentiable Biomechanics Unlocks Opportunities for Markerless Motion Capture](https://mlsanity.com/paper/2402.17192/) (2024)
- [BioPose: Biomechanically-accurate 3D Pose Estimation from Monocular Videos](https://mlsanity.com/paper/2501.07800/) (2025)
- [Improved Trajectory Reconstruction for Markerless Pose Estimation](https://mlsanity.com/paper/2303.02413/) (2023)
- [ATLAS: Decoupling Skeletal and Shape Parameters for Expressive Parametric Human Modeling](https://mlsanity.com/paper/2508.15767/) (2025)
- [PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research](https://mlsanity.com/paper/2203.08792/) (2022)
- [MHR: Momentum Human Rig](https://mlsanity.com/paper/2511.15586/) (2025)
- [Differentiable Biomechanics for Markerless Motion Capture in Upper Limb Stroke Rehabilitation: A Comparison with Optical Motion Capture](https://mlsanity.com/paper/2411.14992/) (2024)
- [Portable Biomechanics Laboratory: Clinically Accessible Movement Analysis from a Handheld Smartphone](https://mlsanity.com/paper/2507.08268/) (2025)
- [Calibrated Uncertainty for Trustworthy Clinical Gait Analysis Using Probabilistic Multiview Markerless Motion Capture](https://mlsanity.com/paper/2601.22412/) (2026)
- [Biomechanical Reconstruction with Confidence Intervals from Multiview Markerless Motion Capture](https://mlsanity.com/paper/2502.06486/) (2025)

## Built on by (most cited first, 5 of 5)

- [Markerless Motion Capture for Biomechanical Whole-Body Kinematic Estimation in Infants](https://mlsanity.com/paper/2605.17120/) (2026)
- [Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model](https://mlsanity.com/paper/2608.29928/) (2026)
- [Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements](https://mlsanity.com/paper/2607.02796/) (2026)
- [Markerless Motion Capture in Routine Clinical Upper Limb Assessments: Validity and Insights Beyond Ordinal Scoring](https://mlsanity.com/paper/2607.23608/) (2026)
- [Cross-Model Distillation of a Human-Pose Foundation Model from Unannotated Infant Video for Markerless 3D Pose Estimation](https://mlsanity.com/paper/2609.01840/) (2026)

## Abstract (arXiv)

Accurate hand and finger tracking from video has significant clinical applications for monitoring activities of daily living and measuring range of motion, yet monocular video approaches for obtaining hand biomechanics remain under-developed. We present a method that combines the SAM 3D Body foundation model with inverse kinematics optimization in a full-body biomechanical model to extract anatomically-constrained finger joint angles from single-view video. We port SAM 3D Body from PyTorch to JAX for integration with MuJoCo-MJX, enabling GPU-accelerated optimization, and develop a novel mapping between the Momentum Human Rig (MHR) outputs and biomechanical model markers. Validation against 8-camera multiview reconstruction on 4,590 frames from 7 participants performing a variety of hand poses and object manipulation tasks shows finger joint angle errors of approximately 10 degrees and hand position errors of approximately 6 mm, after Procrustes alignment. Results were consistent across camera viewpoints and robust to different methods for producing reference values from multiview video. This work extends monocular biomechanical analysis to detailed finger tracking, expanding access to quantitative characterization of hand movement from readily available video.