# UniAP: Towards Universal Animal Perception in Vision via Few-shot Learning

arXiv: https://arxiv.org/abs/2308.09953 · HTML: https://mlsanity.com/paper/2308.09953/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 56 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)
- [ParseNet: Looking Wider to See Better](https://mlsanity.com/paper/1506.04579/) (2015)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2211.14813/) (2022)
- [AP-10K: A Benchmark for Animal Pose Estimation in the Wild](https://mlsanity.com/paper/2108.12617/) (2021)
- [Cross-Domain Adaptation for Animal Pose Estimation](https://mlsanity.com/paper/1908.05806/) (2019)
- [Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding](https://mlsanity.com/paper/2204.08129/) (2022)
- [APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking](https://mlsanity.com/paper/2206.05683/) (2022)
- [From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation](https://mlsanity.com/paper/2103.14843/) (2021)
- [FGN: Fully Guided Network for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2003.13954/) (2020)
- [One-Shot Instance Segmentation](https://mlsanity.com/paper/1811.11507/) (2018)
- [Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models](https://mlsanity.com/paper/2210.15138/) (2022)

## Built on by (most cited first, 7 of 7)

- [X-Pose: Detecting Any Keypoints](https://mlsanity.com/paper/2310.08530/) (2023)
- [AniClipart: Clipart Animation with Text-to-Video Priors](https://mlsanity.com/paper/2404.12347/) (2024)
- [Learning Structure-Supporting Dependencies via Keypoint Interactive Transformer for General Mammal Pose Estimation](https://mlsanity.com/paper/2502.18214/) (2025)
- [UniFS: Universal Few-shot Instance Perception with Point Representations](https://mlsanity.com/paper/2404.19401/) (2024)
- [Recurrent Feature Mining and Keypoint Mixup Padding for Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2503.21140/) (2025)
- [Towards Texture- And Shape-Independent 3D Keypoint Estimation in Birds](https://mlsanity.com/paper/2505.16633/) (2025)
- [Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection](https://mlsanity.com/paper/2507.07994/) (2025)

## Abstract (arXiv)

Animal visual perception is an important technique for automatically monitoring animal health, understanding animal behaviors, and assisting animal-related research. However, it is challenging to design a deep learning-based perception model that can freely adapt to different animals across various perception tasks, due to the varying poses of a large diversity of animals, lacking data on rare species, and the semantic inconsistency of different tasks. We introduce UniAP, a novel Universal Animal Perception model that leverages few-shot learning to enable cross-species perception among various visual tasks. Our proposed model takes support images and labels as prompt guidance for a query image. Images and labels are processed through a Transformer-based encoder and a lightweight label encoder, respectively. Then a matching module is designed for aggregating information between prompt guidance and the query image, followed by a multi-head label decoder to generate outputs for various tasks. By capitalizing on the shared visual characteristics among different animals and tasks, UniAP enables the transfer of knowledge from well-studied species to those with limited labeled data or even unseen species. We demonstrate the effectiveness of UniAP through comprehensive experiments in pose estimation, segmentation, and classification tasks on diverse animal species, showcasing its ability to generalize and adapt to new classes with minimal labeled examples.