# Triple-stream Deep Metric Learning of Great Ape Behavioural Actions

arXiv: https://arxiv.org/abs/2301.02642 · HTML: https://mlsanity.com/paper/2301.02642/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 6 Jan 2023.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Seeing biodiversity: perspectives in machine learning for wildlife conservation](https://mlsanity.com/paper/2110.12951/) (2021)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022)
- [An Image is Worth 16x16 Words, What is a Video Worth?](https://mlsanity.com/paper/2103.13915/) (2021)
- [PyTorch Metric Learning](https://mlsanity.com/paper/2008.09164/) (2020)
- [Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition](https://mlsanity.com/paper/1811.07503/) (2018)
- [Visual Identification of Individual Holstein-Friesian Cattle via Deep Metric Learning](https://mlsanity.com/paper/2006.09205/) (2020)
- [Transferring Dense Pose to Proximal Animal Classes](https://mlsanity.com/paper/2003.00080/) (2020)
- [Visual Recognition of Great Ape Behaviours in the Wild](https://mlsanity.com/paper/2011.10759/) (2020)
- [Manifold Mixup: Better Representations by Interpolating Hidden States](https://mlsanity.com/paper/1806.05236/) (2018)
- [PERF-Net: Pose Empowered RGB-Flow Net](https://mlsanity.com/paper/2009.13087/) (2020)
- [Recognition and 3D Localization of Pedestrian Actions from Monocular Video](https://mlsanity.com/paper/2008.01162/) (2020)

## Built on by (most cited first, 14 of 14)

- [PanAf20K: A Large Video Dataset for Wild Ape Detection and Behaviour Recognition](https://mlsanity.com/paper/2401.13554/) (2024)
- [MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps](https://mlsanity.com/paper/2503.18223/) (2025)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification](https://mlsanity.com/paper/2511.15622/) (2025)
- [ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition](https://mlsanity.com/paper/2404.08937/) (2024)
- [The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition](https://mlsanity.com/paper/2502.21201/) (2025)
- [PriVi: Towards A General-Purpose Video Model For Primate Behavior In The Wild](https://mlsanity.com/paper/2511.09675/) (2025)
- [Deep Visual-Genetic Biometrics for Taxonomic Classification of Rare Species](https://mlsanity.com/paper/2305.06695/) (2023)
- [Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology](https://mlsanity.com/paper/2505.02825/) (2025)
- [Diving with Penguins: Detecting Penguins and their Prey in Animal-borne Underwater Videos via Deep Learning](https://mlsanity.com/paper/2308.07267/) (2023)
- [From Forest to Zoo: Great Ape Behavior Recognition with ChimpBehave](https://mlsanity.com/paper/2405.20025/) (2024)
- [Deep in the Jungle: Towards Automating Chimpanzee Population Estimation](https://mlsanity.com/paper/2601.22917/) (2026)
- [Automating Visual Recognition of Leprosy in Wild Chimpanzees](https://mlsanity.com/paper/2607.17395/) (2026)
- [The PanAf-SBR Dataset: Social Behaviour Recognition for Wild Great Apes](https://mlsanity.com/paper/2607.17399/) (2026)

## Abstract (arXiv)

We propose the first metric learning system for the recognition of great ape behavioural actions. Our proposed triple stream embedding architecture works on camera trap videos taken directly in the wild and demonstrates that the utilisation of an explicit DensePose-C chimpanzee body part segmentation stream effectively complements traditional RGB appearance and optical flow streams. We evaluate system variants with different feature fusion techniques and long-tail recognition approaches. Results and ablations show performance improvements of ~12% in top-1 accuracy over previous results achieved on the PanAf-500 dataset containing 180,000 manually annotated frames across nine behavioural actions. Furthermore, we provide a qualitative analysis of our findings and augment the metric learning system with long-tail recognition techniques showing that average per class accuracy -- critical in the domain -- can be improved by ~23% compared to the literature on that dataset. Finally, since our embedding spaces are constructed as metric, we provide first data-driven visualisations of the great ape behavioural action spaces revealing emerging geometry and topology. We hope that the work sparks further interest in this vital application area of computer vision for the benefit of endangered great apes.