mlsanityData to 14 Oct 2026

Cold Case: The Lost MNIST Digits

arXiv 1905.10498 ↗First posted 25 May 2019cs.LG cs.CV stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy. We trace each MNIST digit to its NIST source and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed, they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing performances. Our results unambiguously confirm the trends observed by Recht et al. [2018, 2019]: although the misclassification rates are slightly off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same digits.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 15; 2023: 11; 2024: 10; 2025: 10; 2026: 4.

62 papers.

Sort:

By area:

  1. 2021Robust fine-tuning of zero-shot modelscited by 797
  2. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  3. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  4. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
  5. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  6. 2021The Benchmark Lotterycited by 136
  7. 2021Predicting with Confidence on Unseen Distributionscited by 116
  8. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  9. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 75
  10. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  11. 2022Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classificationcited by 26
  12. 2021Do Question Answering Modeling Improvements Hold Across Benchmarks?cited by 20
38 more
  1. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  2. 2023Mechanism of feature learning in convolutional neural networkscited by 17
  3. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  4. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 13
  5. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  6. 2022Fix-A-Step: Semi-supervised Learning from Uncurated Unlabeled Datacited by 11
  7. 2022A Measure of the Complexity of Neural Representations based on Partial Information Decompositioncited by 11
  8. 2023DMLR: Data-centric Machine Learning Research -- Past, Present and Futurecited by 11
  9. 2024Model Inversion Attacks: A Survey of Approaches and Countermeasurescited by 11
  10. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  11. 2021Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version)cited by 7
  12. 2022Meta Mirror Descent: Optimiser Learning for Fast Convergencecited by 7
  13. 2023Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulationscited by 7
  14. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  15. 2024Predicting the Performance of Foundation Models via Agreement-on-the-Linecited by 7
  16. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 7
  17. 2022OmniPrint: A Configurable Printed Character Synthesizercited by 5
  18. 2022Decision boundaries and convex hulls in the feature space that deep learning functions learn from imagescited by 5
  19. 2021Extending Contrastive Learning to Unsupervised Coreset Selectioncited by 4
  20. 2021Tensor network to learn the wavefunction of datacited by 4
  21. 2022End-to-end Multiple Instance Learning with Gradient Accumulationcited by 4
  22. 2023Echo of Neighbors: Privacy Amplification for Personalized Private Federated Learning with Shuffle Modelcited by 4
  23. 2021Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Searchcited by 3
  24. 2024Tensor tree learns hidden relational structures in data to construct generative modelscited by 3
  25. 2026Leaderboard Incentives: Model Rankings under Strategic Post-Trainingcited by 3
  26. 2022Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimizationcited by 2
  27. 2024Multiple Instance Verificationcited by 2
  28. 2022Oral cancer detection and interpretation: Deep multiple instance learning versus conventional deep single instance learningcited by 1
  29. 2023Benchmarking Low-Shot Robustness to Natural Distribution Shiftscited by 1
  30. 2023A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifierscited by 1
  31. 2023Evaluation of human-model prediction difference on the Internet Scale of Datacited by 1
  32. 2024Mitigating covariate shift in non-colocated data with learned parameter priorscited by 1
  33. 2025The Sample Complexity of Parameter-Free Stochastic Convex Optimizationcited by 1
  34. 2025From MNIST to ImageNet: Understanding the Scalability Boundaries of Differentiable Logic Gate Networkscited by 1
  35. 2025Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLioncited by 1
  36. 2026Bridging Domain Expertise and Generalization for Performance Estimationcited by 1
  37. 2026Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankingscited by 1
  38. 2021MANDERA: Malicious Node Detection in Federated Learning via Rankingcited by 0

50 most cited of 62 All 62 (JSON)