# Open-Source Tools for Behavioral Video Analysis: Setup, Methods, and Development

arXiv: https://arxiv.org/abs/2204.02842 · HTML: https://mlsanity.com/paper/2204.02842/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 108 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Task Programming: Learning Data Efficient Behavior Representations](https://mlsanity.com/paper/2011.13917/) (2020)
- [Self-Supervised Keypoint Discovery in Behavioral Videos](https://mlsanity.com/paper/2112.05121/) (2021)
- [Mapping the stereotyped behaviour of freely-moving fruit flies](https://mlsanity.com/paper/1310.4249/) (2013)
- [Measuring behavior across scales](https://mlsanity.com/paper/1712.05784/) (2017)
- [Unsupervised Learning of Neurosymbolic Encoders](https://mlsanity.com/paper/2107.13132/) (2021)
- [SimBa: An Efficient Tool for Approximating Rips-filtration Persistence via Simplicial Batch-collapse](https://mlsanity.com/paper/1609.07517/) (2016)

## Built on by (most cited first, 4 of 4)

- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervision](https://mlsanity.com/paper/2409.09455/) (2024)
- [Mouse Lockbox Dataset: Behavior Recognition for Mice Solving Lockboxes](https://mlsanity.com/paper/2505.15408/) (2025)
- [How many classes do we need to see for novel class discovery?](https://mlsanity.com/paper/2509.15585/) (2025)

## Abstract (arXiv)

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional "center of mass" tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.