# Measuring and modeling the motor system with machine learning

arXiv: https://arxiv.org/abs/2103.11775 · HTML: https://mlsanity.com/paper/2103.11775/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 150 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization](https://mlsanity.com/paper/1908.09375/) (2019)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [Pretraining boosts out-of-domain robustness for pose estimation](https://mlsanity.com/paper/1909.11229/) (2019)
- [Deep learning tools for the measurement of animal behavior in neuroscience](https://mlsanity.com/paper/1909.13868/) (2019)
- [Multi-Fiber Networks for Video Recognition](https://mlsanity.com/paper/1807.11195/) (2018)
- [Deep Reinforcement Learning and its Neuroscientific Implications](https://mlsanity.com/paper/2007.03750/) (2020)
- [Artificial neural networks for neuroscientists: A primer](https://mlsanity.com/paper/2006.01001/) (2020)
- [Deep neuroethology of a virtual rodent](https://mlsanity.com/paper/1911.09451/) (2019)
- [Mapping the stereotyped behaviour of freely-moving fruit flies](https://mlsanity.com/paper/1310.4249/) (2013)
- [A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives](https://mlsanity.com/paper/2009.00564/) (2020)
- [Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion](https://mlsanity.com/paper/1912.05328/) (2019)
- [Toward Goal-Driven Neural Network Models for the Rodent Whisker-Trigeminal System](https://mlsanity.com/paper/1706.07555/) (2017)
- [Deep Reinforcement Learning for Neural Control](https://mlsanity.com/paper/2006.07352/) (2020)

## Built on by (most cited first, 7 of 7)

- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [SuperAnimal pretrained pose estimation models for behavioral analysis](https://mlsanity.com/paper/2203.07436/) (2022)
- [AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild](https://mlsanity.com/paper/2103.13282/) (2021)
- [Latent Exploration for Reinforcement Learning](https://mlsanity.com/paper/2305.20065/) (2023)
- [A Rubric for Human-like Agents and NeuroAI](https://mlsanity.com/paper/2212.04401/) (2022)
- [Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems](https://mlsanity.com/paper/2411.15234/) (2024)
- [Machine Learning in Biomechanics: Key Applications and Limitations in Walking, Running, and Sports Movements](https://mlsanity.com/paper/2503.03717/) (2025)

## Abstract (arXiv)

The utility of machine learning in understanding the motor system is promising a revolution in how to collect, measure, and analyze data. The field of movement science already elegantly incorporates theory and engineering principles to guide experimental work, and in this review we discuss the growing use of machine learning: from pose estimation, kinematic analyses, dimensionality reduction, and closed-loop feedback, to its use in understanding neural correlates and untangling sensorimotor systems. We also give our perspective on new avenues where markerless motion capture combined with biomechanical modeling and neural networks could be a new platform for hypothesis-driven research.