# Markerless Motion Capture and Biomechanical Analysis Pipeline

arXiv: https://arxiv.org/abs/2303.10654 · HTML: https://mlsanity.com/paper/2303.10654/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Whole-Body Human Pose Estimation in the Wild](https://mlsanity.com/paper/2007.11858/) (2020)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats](https://mlsanity.com/paper/2212.14474/) (2022)
- [MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation](https://mlsanity.com/paper/2007.07227/) (2020)
- [Improved Trajectory Reconstruction for Markerless Pose Estimation](https://mlsanity.com/paper/2303.02413/) (2023)
- [PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research](https://mlsanity.com/paper/2203.08792/) (2022)
- [On Triangulation as a Form of Self-Supervision for 3D Human Pose Estimation](https://mlsanity.com/paper/2203.15865/) (2022)

## Built on by (most cited first, 10 of 10)

- [Differentiable Biomechanics Unlocks Opportunities for Markerless Motion Capture](https://mlsanity.com/paper/2402.17192/) (2024)
- [Portable Biomechanics Laboratory: Clinically Accessible Movement Analysis from a Handheld Smartphone](https://mlsanity.com/paper/2507.08268/) (2025)
- [Self-Supervised Learning of Gait-Based Biomarkers](https://mlsanity.com/paper/2307.16321/) (2023)
- [Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation](https://mlsanity.com/paper/2402.12676/) (2024)
- [Dynamic Gaussian Splatting from Markerless Motion Capture can Reconstruct Infants Movements](https://mlsanity.com/paper/2310.19441/) (2023)
- [BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds](https://mlsanity.com/paper/2503.00389/) (2025)
- [Markerless Motion Capture for Biomechanical Whole-Body Kinematic Estimation in Infants](https://mlsanity.com/paper/2605.17120/) (2026)
- [Personalization of Wearable Sensor-Based Joint Kinematic Estimation Using Computer Vision for Hip Exoskeleton Applications](https://mlsanity.com/paper/2411.15366/) (2024)
- [DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows](https://mlsanity.com/paper/2602.16585/) (2026)
- [Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements](https://mlsanity.com/paper/2607.02796/) (2026)

## Abstract (arXiv)

Markerless motion capture using computer vision and human pose estimation (HPE) has the potential to expand access to precise movement analysis. This could greatly benefit rehabilitation by enabling more accurate tracking of outcomes and providing more sensitive tools for research. There are numerous steps between obtaining videos to extracting accurate biomechanical results and limited research to guide many critical design decisions in these pipelines. In this work, we analyze several of these steps including the algorithm used to detect keypoints and the keypoint set, the approach to reconstructing trajectories for biomechanical inverse kinematics and optimizing the IK process. Several features we find important are: 1) using a recent algorithm trained on many datasets that produces a dense set of biomechanically-motivated keypoints, 2) using an implicit representation to reconstruct smooth, anatomically constrained marker trajectories for IK, 3) iteratively optimizing the biomechanical model to match the dense markers, 4) appropriate regularization of the IK process. Our pipeline makes it easy to obtain accurate biomechanical estimates of movement in a rehabilitation hospital.