# Bugs in the Data: How ImageNet Misrepresents Biodiversity

arXiv: https://arxiv.org/abs/2208.11695 · HTML: https://mlsanity.com/paper/2208.11695/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [From ImageNet to Image Classification: Contextualizing Progress on Benchmarks](https://mlsanity.com/paper/2005.11295/) (2020)
- [The Devil is in the Tails: Fine-grained Classification in the Wild](https://mlsanity.com/paper/1709.01450/) (2017)
- [Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development](https://mlsanity.com/paper/2108.04308/) (2021)
- [A Study of Face Obfuscation in ImageNet](https://mlsanity.com/paper/2103.06191/) (2021)
- [The iWildCam 2021 Competition Dataset](https://mlsanity.com/paper/2105.03494/) (2021)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Bringing the People Back In: Contesting Benchmark Machine Learning Datasets](https://mlsanity.com/paper/2007.07399/) (2020)
- [The iWildCam 2018 Challenge Dataset](https://mlsanity.com/paper/1904.05986/) (2019)

## Built on by (most cited first, 24 of 24)

- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality](https://mlsanity.com/paper/2211.00768/) (2022)
- [ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification](https://mlsanity.com/paper/2304.05538/) (2023)
- [Application-Driven Innovation in Machine Learning](https://mlsanity.com/paper/2403.17381/) (2024)
- [BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity](https://mlsanity.com/paper/2406.17720/) (2024)
- [Measuring Data](https://mlsanity.com/paper/2212.05129/) (2022)
- [Understanding the Detrimental Class-level Effects of Data Augmentation](https://mlsanity.com/paper/2401.01764/) (2023)
- [Flaws of ImageNet, Computer Vision's Favourite Dataset](https://mlsanity.com/paper/2412.00076/) (2024)
- [Invariant Learning via Diffusion Dreamed Distribution Shifts](https://mlsanity.com/paper/2211.10370/) (2022)
- [AnuraSet: A dataset for benchmarking Neotropical anuran calls identification in passive acoustic monitoring](https://mlsanity.com/paper/2307.06860/) (2023)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Insect Identification in the Wild: The AMI Dataset](https://mlsanity.com/paper/2406.12452/) (2024)
- [Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization](https://mlsanity.com/paper/2504.18026/) (2025)
- [Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs](https://mlsanity.com/paper/2405.18740/) (2024)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification](https://mlsanity.com/paper/2410.05057/) (2024)
- [Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring](https://mlsanity.com/paper/2503.01691/) (2025)
- [Reflections from the Workshop on AI-Assisted Decision Making for Conservation](https://mlsanity.com/paper/2307.08774/) (2023)
- [Leveraging Human-Machine Interactions for Computer Vision Dataset Quality Enhancement](https://mlsanity.com/paper/2401.17736/) (2024)
- [Multimodal Large Language Models as Image Classifiers](https://mlsanity.com/paper/2603.06578/) (2026)
- [The Impact of the Single-Label Assumption in Image Recognition Benchmarking](https://mlsanity.com/paper/2412.18409/) (2024)
- [Unification of popular artificial neural network activation functions](https://mlsanity.com/paper/2302.11007/) (2023)
- [Ethosight: A Reasoning-Guided Iterative Learning System for Nuanced Perception based on Joint-Embedding & Contextual Label Affinity](https://mlsanity.com/paper/2307.10577/) (2023)
- [CrypticBio: A Large Multimodal Dataset for Visually Confusing Biodiversity](https://mlsanity.com/paper/2505.14707/) (2025)

## Abstract (arXiv)

ImageNet-1k is a dataset often used for benchmarking machine learning (ML) models and evaluating tasks such as image recognition and object detection. Wild animals make up 27% of ImageNet-1k but, unlike classes representing people and objects, these data have not been closely scrutinized. In the current paper, we analyze the 13,450 images from 269 classes that represent wild animals in the ImageNet-1k validation set, with the participation of expert ecologists. We find that many of the classes are ill-defined or overlapping, and that 12% of the images are incorrectly labeled, with some classes having >90% of images incorrect. We also find that both the wildlife-related labels and images included in ImageNet-1k present significant geographical and cultural biases, as well as ambiguities such as artificial animals, multiple species in the same image, or the presence of humans. Our findings highlight serious issues with the extensive use of this dataset for evaluating ML systems, the use of such algorithms in wildlife-related tasks, and more broadly the ways in which ML datasets are commonly created and curated.