# 3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data

arXiv: https://arxiv.org/abs/2402.13172 · HTML: https://mlsanity.com/paper/2402.13172/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 18 of 18)

- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2203.00859/) (2022)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints](https://mlsanity.com/paper/2003.09282/) (2020)
- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [Direct Multi-view Multi-person 3D Pose Estimation](https://mlsanity.com/paper/2111.04076/) (2021)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images](https://mlsanity.com/paper/2109.05885/) (2021)
- [Learned Vertex Descent: A New Direction for 3D Human Model Fitting](https://mlsanity.com/paper/2205.06254/) (2022)
- [OSSO: Obtaining Skeletal Shape from Outside](https://mlsanity.com/paper/2204.10129/) (2022)
- [POCO: 3D Pose and Shape Estimation with Confidence](https://mlsanity.com/paper/2308.12965/) (2023)
- [Towards Single Camera Human 3D-Kinematics](https://mlsanity.com/paper/2301.05435/) (2023)
- [Deep learning-based estimation of whole-body kinematics from multi-view images](https://mlsanity.com/paper/2307.05896/) (2023)

## Built on by (most cited first, 2 of 2)

- [Reconstructing Humans with a Biomechanically Accurate Skeleton](https://mlsanity.com/paper/2503.21751/) (2025)
- [BioHuman: Learning Biomechanical Human Representations from Video](https://mlsanity.com/paper/2605.14772/) (2026)

## Abstract (arXiv)

Accurate 3D kinematics estimation of human body is crucial in various applications for human health and mobility, such as rehabilitation, injury prevention, and diagnosis, as it helps to understand the biomechanical loading experienced during movement. Conventional marker-based motion capture is expensive in terms of financial investment, time, and the expertise required. Moreover, due to the scarcity of datasets with accurate annotations, existing markerless motion capture methods suffer from challenges including unreliable 2D keypoint detection, limited anatomic accuracy, and low generalization capability. In this work, we propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views with consideration of biomechanical prior and spatio-temporal information. To train the model, we create synthetic dataset ODAH with accurate kinematics annotations generated by aligning the body mesh from the SMPL-X model and a full-body OpenSim skeletal model. Our extensive experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods when evaluated across multiple datasets, revealing a promising direction for enhancing video-based human motion capture