# Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study

arXiv: https://arxiv.org/abs/2602.03894 · HTML: https://mlsanity.com/paper/2602.03894/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Efficient Pipeline for Camera Trap Image Review](https://mlsanity.com/paper/1907.06772/) (2019)
- [A deep active learning system for species identification and counting in camera trap images](https://mlsanity.com/paper/1910.09716/) (2019)
- [Focus on the Positives: Self-Supervised Learning for Biodiversity Monitoring](https://mlsanity.com/paper/2108.06435/) (2021)
- [Towards Zero-Shot Camera Trap Image Categorization](https://mlsanity.com/paper/2410.12769/) (2024)
- [Hierarchical Re-Classification: Combining Animal Classification Models with Vision Transformers](https://mlsanity.com/paper/2510.14594/) (2025)

## Built on by (most cited first, 5 of 5)

- [Finding Needles in the Haystack: Transductive Active Labeling in Ecology](https://mlsanity.com/paper/2606.03821/) (2026)
- [Oh Deer, How Should I Handle This? Seasonal Priors for Selective Wildlife Annotation and Classification](https://mlsanity.com/paper/2608.02762/) (2026)
- [Label-efficient underwater image classification with logistic regression on frozen foundation model embeddings](https://mlsanity.com/paper/2604.00313/) (2026)
- [Inference-Path Optimization via Circuit Duplication in Frozen Visual Transformers for Marine Species Classification](https://mlsanity.com/paper/2604.03428/) (2026)
- [trACT: temporal revelation Airborne Camera Trap](https://mlsanity.com/paper/2610.09417/) (2026)

## Abstract (arXiv)

Manual labeling of animal images remains a significant bottleneck in ecological research, limiting the scale and efficiency of biodiversity monitoring efforts. This study investigates whether state-of-the-art Vision Transformer (ViT) foundation models can reduce thousands of unlabeled animal images directly to species-level clusters. We present a comprehensive benchmarking framework evaluating five ViT models combined with five dimensionality reduction techniques and four clustering algorithms, two supervised and two unsupervised, across 60 species (30 mammals and 30 birds), with each test using a random subset of 200 validated images per species. We investigate when clustering succeeds at species-level, where it fails, and whether clustering within the species-level reveals ecologically meaningful patterns such as sex, age, or phenotypic variation. Our results demonstrate near-perfect species-level clustering (V-measure: 0.958) using DINOv3 embeddings with t-SNE and supervised hierarchical clustering methods. Unsupervised approaches achieve competitive performance (0.943) while requiring no prior species knowledge, rejecting only 1.14% of images as outliers requiring expert review. We further demonstrate robustness to realistic long-tailed distributions of species and show that intentional over-clustering can reliably extract intra-specific variation including age classes, sexual dimorphism, and pelage differences. We introduce an open-source benchmarking toolkit and provide recommendations for ecologists to select appropriate methods for sorting their specific taxonomic groups and data.