# Markerless tracking of user-defined features with deep learning

arXiv: https://arxiv.org/abs/1804.03142 · HTML: https://mlsanity.com/paper/1804.03142/

## Key facts

- Cited by 206 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2018.

## Built on by (most cited first, 30 of 206)

- [Seeing biodiversity: perspectives in machine learning for wildlife conservation](https://mlsanity.com/paper/2110.12951/) (2021)
- [Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding](https://mlsanity.com/paper/2204.08129/) (2022)
- [Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution](https://mlsanity.com/paper/2210.08340/) (2022)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [SuperAnimal pretrained pose estimation models for behavioral analysis](https://mlsanity.com/paper/2203.07436/) (2022)
- [Dynamic Point Fields](https://mlsanity.com/paper/2304.02626/) (2023)
- [Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction](https://mlsanity.com/paper/2102.10106/) (2021)
- [The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions](https://mlsanity.com/paper/2104.02710/) (2021)
- [End-to-End Trainable Multi-Instance Pose Estimation with Transformers](https://mlsanity.com/paper/2103.12115/) (2021)
- [AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild](https://mlsanity.com/paper/2103.13282/) (2021)
- [From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans](https://mlsanity.com/paper/2509.06607/) (2025)
- [ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors](https://mlsanity.com/paper/2310.16447/) (2023)
- [Self-Supervised Keypoint Discovery in Behavioral Videos](https://mlsanity.com/paper/2112.05121/) (2021)
- [X-Pose: Detecting Any Keypoints](https://mlsanity.com/paper/2310.08530/) (2023)
- [AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer](https://mlsanity.com/paper/2412.00837/) (2024)
- [Neural Encoding and Decoding at Scale](https://mlsanity.com/paper/2504.08201/) (2025)
- [hSMAL: Detailed Horse Shape and Pose Reconstruction for Motion Pattern Recognition](https://mlsanity.com/paper/2106.10102/) (2021)
- [Global Instance Tracking: Locating Target More Like Humans](https://mlsanity.com/paper/2202.13073/) (2022)
- [3D-POP -- An automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture](https://mlsanity.com/paper/2303.13174/) (2023)
- [AniClipart: Clipart Animation with Text-to-Video Priors](https://mlsanity.com/paper/2404.12347/) (2024)
- [Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting](https://mlsanity.com/paper/2304.00933/) (2023)
- [SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking](https://mlsanity.com/paper/2502.11534/) (2025)
- [Semi-supervised Keypoint Localization](https://mlsanity.com/paper/2101.07988/) (2021)
- [T-LEAP: Occlusion-robust pose estimation of walking cows using temporal information](https://mlsanity.com/paper/2104.08029/) (2021)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [Markerless human pose estimation for biomedical applications: a survey](https://mlsanity.com/paper/2308.00519/) (2023)
- [AmadeusGPT: a natural language interface for interactive animal behavioral analysis](https://mlsanity.com/paper/2307.04858/) (2023)
- [3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking](https://mlsanity.com/paper/2308.15316/) (2023)
- [Machine Learning and Deep Learning -- A review for Ecologists](https://mlsanity.com/paper/2204.05023/) (2022)
- [Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks](https://mlsanity.com/paper/2103.15449/) (2021)

## Abstract (arXiv)

Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, yet markers are intrusive (especially for smaller animals), and the number and location of the markers must be determined a priori. Here, we present a highly efficient method for markerless tracking based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in a broad collection of experimental settings: mice odor trail-tracking, egg-laying behavior in drosophila, and mouse hand articulation in a skilled forelimb task. For example, during the skilled reaching behavior, individual joints can be automatically tracked (and a confidence score is reported). Remarkably, even when a small number of frames are labeled (≈ 200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.