# Doomed to Re-Annotate, Forever: The ImageNet Story

arXiv: https://arxiv.org/abs/2608.13783 · HTML: https://mlsanity.com/paper/2608.13783/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 48 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](https://mlsanity.com/paper/2407.08083/) (2024)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [From ImageNet to Image Classification: Contextualizing Progress on Benchmarks](https://mlsanity.com/paper/2005.11295/) (2020)
- [When does dough become a bagel? Analyzing the remaining mistakes on ImageNet](https://mlsanity.com/paper/2205.04596/) (2022)
- [From Pixels to Prose: A Large Dataset of Dense Image Captions](https://mlsanity.com/paper/2406.10328/) (2024)

## Built on by (most cited first, 1 of 1)

- [Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Set](https://mlsanity.com/paper/2609.01027/) (2026)

## Abstract (arXiv)

Top-1 accuracy on ImageNet-1k remains the most commonly reported metric in visual recognition. Quality issues with the dataset have been repeatedly reported, yet the original 2012 noisy labels are still predominantly used. The paper presents a comprehensive effort, which goes well beyond prior correction attempts, towards obtaining accurate and complete ImageNet-1k validation set annotations. The result, ReImageNet, includes multilabel correction, object localization, revised class definitions, and semantic attributes (text-recognition, rendition, reflection, crowd, dominant). The reannotation reveals that approximately 12% of the original ImageNet-1k labels are incorrect, 33.3% of images are multilabel and 3.8% contain no object from an ImageNet-1k class. With the new labels, top-1 accuracy increases by up to 1.2% for supervised models and by 5-6% for MLLMs. We argue that annotation at ImageNet scale cannot realistically be completed in one pass, as errors and definitional issues are discovered only through annotating, and we build our pipeline around repeated refinement and error checking. We observed that human and LLM collaboration with appropriate tooling represents the current quality ceiling for annotation at this scale. ImageNet-1k issues propagate into its derivative test sets, indicating that the problem is structural rather than specific to any single benchmark. All annotations, class definitions, guidelines, and analysis code have been publicly released. Project page: https://vrg.fel.cvut.cz/reimagenet Annotations: https://huggingface.co/datasets/vrg-prague/ReImageNet Code: https://github.com/klarajanouskova/ImageNet