# Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology

arXiv: https://arxiv.org/abs/2505.02825 · HTML: https://mlsanity.com/paper/2505.02825/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 5 May 2025.

## Builds on (most cited first, 21 of 21)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Seeing biodiversity: perspectives in machine learning for wildlife conservation](https://mlsanity.com/paper/2110.12951/) (2021)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding](https://mlsanity.com/paper/2204.08129/) (2022)
- [Deep learning tools for the measurement of animal behavior in neuroscience](https://mlsanity.com/paper/1909.13868/) (2019)
- [MammalNet: A Large-scale Video Benchmark for Mammal Recognition and Behavior Understanding](https://mlsanity.com/paper/2306.00576/) (2023)
- [Application-Driven Innovation in Machine Learning](https://mlsanity.com/paper/2403.17381/) (2024)
- [PanAf20K: A Large Video Dataset for Wild Ape Detection and Behaviour Recognition](https://mlsanity.com/paper/2401.13554/) (2024)
- [3D-POP -- An automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture](https://mlsanity.com/paper/2303.13174/) (2023)
- [Triple-stream Deep Metric Learning of Great Ape Behavioural Actions](https://mlsanity.com/paper/2301.02642/) (2023)
- [BuckTales : A multi-UAV dataset for multi-object tracking and re-identification of wild antelopes](https://mlsanity.com/paper/2411.06896/) (2024)
- [3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking](https://mlsanity.com/paper/2308.15316/) (2023)
- [Automated Distance Estimation for Wildlife Camera Trapping](https://mlsanity.com/paper/2202.04613/) (2022)
- [Overcoming the Distance Estimation Bottleneck in Estimating Animal Abundance with Camera Traps](https://mlsanity.com/paper/2105.04244/) (2021)

## Built on by (most cited first, 1 of 1)

- [CHIRP dataset: towards long-term, individual-level, behavioral monitoring of bird populations in the wild](https://mlsanity.com/paper/2603.25524/) (2026)

## Abstract (arXiv)

Computer vision methods have demonstrated considerable potential to streamline ecological and biological workflows, with a growing number of datasets and models becoming available to the research community. However, these resources focus predominantly on evaluation using machine learning metrics, with relatively little emphasis on how their application impacts downstream analysis. We argue that models should be evaluated using application-specific metrics that directly represent model performance in the context of its final use case. To support this argument, we present two disparate case studies: (1) estimating chimpanzee abundance and density with camera trap distance sampling when using a video-based behaviour classifier and (2) estimating head rotation in pigeons using a 3D posture estimator. We show that even models with strong machine learning performance (e.g., 87% mAP) can yield data that leads to discrepancies in abundance estimates compared to expert-derived data. Similarly, the highest-performing models for posture estimation do not produce the most accurate inferences of gaze direction in pigeons. Motivated by these findings, we call for researchers to integrate application-specific metrics in ecological/biological datasets, allowing for models to be benchmarked in the context of their downstream application and to facilitate better integration of models into application workflows.