# A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

arXiv: https://arxiv.org/abs/2009.00564 · HTML: https://mlsanity.com/paper/2009.00564/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2020.

## Built on by (most cited first, 18 of 18)

- [Seeing biodiversity: perspectives in machine learning for wildlife conservation](https://mlsanity.com/paper/2110.12951/) (2021)
- [SuperAnimal pretrained pose estimation models for behavioral analysis](https://mlsanity.com/paper/2203.07436/) (2022)
- [Perspectives on individual animal identification from biology and computer vision](https://mlsanity.com/paper/2103.00560/) (2021)
- [End-to-End Trainable Multi-Instance Pose Estimation with Transformers](https://mlsanity.com/paper/2103.12115/) (2021)
- [AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild](https://mlsanity.com/paper/2103.13282/) (2021)
- [Transflower: probabilistic autoregressive dance generation with multimodal attention](https://mlsanity.com/paper/2106.13871/) (2021)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [Measuring and modeling the motor system with machine learning](https://mlsanity.com/paper/2103.11775/) (2021)
- [Few-shot Geometry-Aware Keypoint Localization](https://mlsanity.com/paper/2303.17216/) (2023)
- [Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories](https://mlsanity.com/paper/2406.03625/) (2024)
- [Automated Behavioral Analysis Using Instance Segmentation](https://mlsanity.com/paper/2312.07723/) (2023)
- [Deep-Learning-Based Markerless Pose Estimation Systems in Gait Analysis: DeepLabCut Custom Training and the Refinement Function](https://mlsanity.com/paper/2407.10590/) (2024)
- [BlanketGen - A synthetic blanket occlusion augmentation pipeline for MoCap datasets](https://mlsanity.com/paper/2210.12035/) (2022)
- [Diffusion Models Enable Zero-Shot Pose Estimation for Lower-Limb Prosthetic Users](https://mlsanity.com/paper/2312.07854/) (2023)
- [A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets](https://mlsanity.com/paper/2201.02503/) (2022)
- [Machine Learning for Optical Motion Capture-driven Musculoskeletal Modelling from Inertial Motion Capture Data](https://mlsanity.com/paper/2209.14456/) (2022)
- [SuDA: Support-based Domain Adaptation for Sim2Real Motion Capture with Flexible Sensors](https://mlsanity.com/paper/2405.16152/) (2024)
- [TrackStudio: An Integrated Toolkit for Markerless Tracking](https://mlsanity.com/paper/2511.07624/) (2025)

## Abstract (arXiv)

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.