mlsanityData to 7 Oct 2026

Do CIFAR-10 Classifiers Generalize to CIFAR-10?

arXiv 1806.00451 ↗First posted 1 Jun 2018cs.LG stat.MLcited by 197

Not on the map. Why?

Abstract (arXiv)

Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images. Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even minute natural variations in the data distribution.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 41; 2023: 54; 2024: 31; 2025: 28; 2026: 15.

197 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  3. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  6. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  7. 2021The Benchmark Lotterycited by 133
  8. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  9. 2021Epistemic Neural Networkscited by 120
  10. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  11. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  12. 2021On Interaction Between Augmentations and Corruptions in Natural Corruption Robustnesscited by 98
38 more
  1. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  2. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  3. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  4. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  5. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  6. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  7. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  8. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  9. 2023On Pitfalls of Test-Time Adaptationcited by 63
  10. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  11. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  12. 2022GraphWorld: Fake Graphs Bring Real Insights for GNNscited by 56
  13. 2024Beyond Model Adaptation at Test Time: A Surveycited by 55
  14. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  15. 2023Multimodal LLMs for health grounded in individual-specific datacited by 48
  16. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  17. 2023In Search of Lost Online Test-time Adaptation: A Surveycited by 43
  18. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  19. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  20. 2023On the Robustness of Open-World Test-Time Training: Self-Training with Dynamic Prototype Expansioncited by 34
  21. 2022TTTFlow: Unsupervised Test-Time Training with Normalizing Flowcited by 33
  22. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  23. 2022Sparse Visual Counterfactual Explanations in Image Spacecited by 30
  24. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  25. 2023Frequency Domain-based Dataset Distillationcited by 28
  26. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  27. 2021Robust Object Detection via Instance-Level Temporal Cycle Confusioncited by 24
  28. 2022Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classificationcited by 24
  29. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  30. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  31. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  32. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  33. 2023ClusT3: Information Invariant Test-Time Trainingcited by 21
  34. 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
  35. 2023Function-Space Regularization in Neural Networks: A Probabilistic Perspectivecited by 20
  36. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Modelscited by 20
  37. 2022A Learning Based Hypothesis Test for Harmful Covariate Shiftcited by 18
  38. 2023Pixel-Grounded Prototypical Part Networkscited by 18

50 most cited of 197 All 197 (JSON)