# A Universal Action Space for General Behavior Analysis

arXiv: https://arxiv.org/abs/2602.09518 · HTML: https://mlsanity.com/paper/2602.09518/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding](https://mlsanity.com/paper/2204.08129/) (2022)
- [Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors](https://mlsanity.com/paper/1505.04868/) (2015)
- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [MammalNet: A Large-scale Video Benchmark for Mammal Recognition and Behavior Understanding](https://mlsanity.com/paper/2306.00576/) (2023)
- [TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition](https://mlsanity.com/paper/1703.10667/) (2017)
- [Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition](https://mlsanity.com/paper/1704.01194/) (2017)
- [Deep Learning-based Animal Behavior Analysis: Insights from Mouse Chronic Pain Models](https://mlsanity.com/paper/2508.05138/) (2025)

## Abstract (arXiv)

Analyzing animal and human behavior has long been a challenging task in computer vision. Early approaches from the 1970s to the 1990s relied on hand-crafted edge detection, segmentation, and low-level features such as color, shape, and texture to locate objects and infer their identities-an inherently ill-posed problem. Behavior analysis in this era typically proceeded by tracking identified objects over time and modeling their trajectories using sparse feature points, which further limited robustness and generalization. A major shift occurred with the introduction of ImageNet by Deng and Li in 2010, which enabled large-scale visual recognition through deep neural networks and effectively served as a comprehensive visual dictionary. This development allowed object recognition to move beyond complex low-level processing toward learned high-level representations. In this work, we follow this paradigm to build a large-scale Universal Action Space (UAS) using existing labeled human-action datasets. We then use this UAS as the foundation for analyzing and categorizing mammalian and chimpanzee behavior datasets. The source code is released on GitHub at https://github.com/franktpmvu/Universal-Action-Space.