mlsanityData to 7 Oct 2026

From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

arXiv 2005.11295 ↗First posted 22 May 2020cs.CV cs.LG stat.MLcited by 90

Not on the map. Why?

Abstract (arXiv)

Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the consequences of employing such a pipeline, focusing on the popular ImageNet dataset. We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset---including the introduction of biases that state-of-the-art models exploit. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for. Finally, our findings emphasize the need to augment our current model training and evaluation toolkit to take such misalignments into account. To facilitate further research, we release our refined ImageNet annotations at https://github.com/MadryLab/ImageNetMultiLabel.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 22; 2023: 15; 2024: 6; 2025: 13; 2026: 7.

90 papers.

Sort:

By area:

  1. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  2. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  3. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  4. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  5. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  6. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  7. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  8. 2023LAVA: Data Valuation without Pre-Specified Learning Algorithmscited by 69
  9. 2021Editing a classifier by rewriting its prediction rulescited by 64
  10. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 60
  11. 2022Hyperbolic Contrastive Learning for Visual Representations beyond Objectscited by 58
  12. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
38 more
  1. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  2. 2022Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNetcited by 41
  3. 2022Adaptive Testing of Computer Vision Modelscited by 39
  4. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  5. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  6. 2023ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classificationcited by 33
  7. 2021Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behaviorcited by 29
  8. 2021How Well do Feature Visualizations Support Causal Understanding of CNN Activations?cited by 28
  9. 2021Active Learning at the ImageNet Scalecited by 28
  10. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  11. 2021FathomNet: A global image database for enabling artificial intelligence in the oceancited by 25
  12. 2022Acknowledging the Unknown for Multi-label Learning with Single Positive Labelscited by 24
  13. 2022Bugs in the Data: How ImageNet Misrepresents Biodiversitycited by 24
  14. 2022Few-shot Learning with Noisy Labelscited by 23
  15. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
  16. 2021A Tale Of Two Long Tailscited by 19
  17. 2021Multi-label Iterated Learning for Image Classification with Label Ambiguitycited by 19
  18. 2023ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracycited by 19
  19. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  20. 2022Core Risk Minimization using Salient ImageNetcited by 15
  21. 2022DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systemscited by 14
  22. 2024ImageNot: A contrast with ImageNet preserves model rankingscited by 14
  23. 2021Did the Model Change? Efficiently Assessing Machine Learning API Shiftscited by 13
  24. 2021LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Trainingcited by 13
  25. 2023Understanding the Detrimental Class-level Effects of Data Augmentationcited by 12
  26. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  27. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 12
  28. 2021Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivitycited by 11
  29. 2022Fix-A-Step: Semi-supervised Learning from Uncurated Unlabeled Datacited by 11
  30. 2021When and How to Fool Explainable Models (and Humans) with Adversarial Examplescited by 10
  31. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  32. 2021Feature Generation for Long-tail Classificationcited by 9
  33. 2023PRIME: Prioritizing Interpretability in Failure Mode Extractioncited by 9
  34. 2021Disentangled Variational Information Bottleneck for Multiview Representation Learningcited by 8
  35. 2021Disentanglement and Generalization Under Correlation Shiftscited by 8
  36. 2022Aligning Visual and Lexical Semanticscited by 7
  37. 2023A semantics-driven methodology for high-quality image annotationcited by 6
  38. 2023Diversity and Diffusion: Observations on Synthetic Image Distributions with Stable Diffusioncited by 6

50 most cited of 90 All 90 (JSON)