Data to 7 Oct 2026

MNIST-C: A Robustness Benchmark for Computer Vision

arXiv 1906.02337 ↗First posted 5 Jun 2019cs.CV cs.LGcited by 137

Not on the map. Why?

Abstract (arXiv)

We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions applied to the MNIST test set, for benchmarking out-of-distribution robustness in computer vision. Through several experiments and visualizations we demonstrate that our corruptions significantly degrade performance of state-of-the-art computer vision models while preserving the semantic content of the test images. In contrast to the popular notion of adversarial robustness, our model-agnostic corruptions do not seek worst-case performance but are instead designed to be broad and diverse, capturing multiple failure modes of modern models. In fact, we find that several previously published adversarial defenses significantly degrade robustness as measured by MNIST-C. We hope that our benchmark serves as a useful tool for future work in designing systems that are able to learn robust feature representations that capture the underlying semantics of the input.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 32; 2023: 31; 2024: 17; 2025: 21; 2026: 13.

137 papers.

Sort:

By area:

2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
2022ADBench: Anomaly Detection Benchmarkcited by 230
2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 150
2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
2021Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformationcited by 67
2023SoTTA: Robust Test-Time Adaptation on Noisy Data Streamscited by 63
2021Visual Representation Learning Does Not Generalize Strongly Within the Same Domaincited by 60
2023Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detectioncited by 57
2021Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restorationcited by 50
2023Credal Bayesian Deep Learningcited by 45

38 more

2023Neither hype nor gloom do DNNs justicecited by 40
2021Provable Repair of Deep Neural Networkscited by 39
2021Tracking the risk of a deployed model and detecting harmful distribution shiftscited by 38
2023Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalizationcited by 34
2023A Survey on Knowledge Editing of Neural Networkscited by 34
2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
2022Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)cited by 27
2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
2024Uncertainty Estimation by Density Aware Evidential Deep Learningcited by 22
2021The Impact of Reinitialization on Generalization in Convolutional Neural Networkscited by 20
2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
2022Improving Diversity with Adversarially Learned Transformations for Domain Generalizationcited by 16
2022Active Multi-Task Representation Learningcited by 13
2024CTBENCH: A Library and Benchmark for Certified Trainingcited by 13
2025WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingalescited by 13
2023HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustnesscited by 12
2024Visual Robustness Benchmark for Visual Question Answering (VQA)cited by 12
2021Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoringcited by 11
2021Exposing Previously Undetectable Faults in Deep Neural Networkscited by 11
2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
2024Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AIcited by 11
2024TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regressioncited by 11
2021SyReNN: A Tool for Analyzing Deep Neural Networkscited by 10
2021A Review and Refinement of Surprise Adequacycited by 9
2021Improving Model Robustness by Adaptively Correcting Perturbation Levels with Active Queriescited by 9
2022Confidence-aware Training of Smoothed Classifiers for Certified Robustnesscited by 9
2023Architecture-Preserving Provable Repair of Deep Neural Networkscited by 9
2023Improved Active Multi-Task Representation Learning via Lassocited by 9
2022CARLA-GeAR: a Dataset Generator for a Systematic Evaluation of Adversarial Robustness of Vision Modelscited by 8
2022Understanding the effect of sparsity on neural networks robustnesscited by 8
2022Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testingcited by 8
2021Drop Clause: Enhancing Performance, Interpretability and Robustness of the Tsetlin Machinecited by 7
2022Fourier-Based Augmentations for Improved Robustness and Uncertainty Calibrationcited by 7
2023Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustnesscited by 7
2021Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Acceleratorscited by 6
2022How Robust are Discriminatively Trained Zero-Shot Learning Models?cited by 6
2021Grid Cell Path Integration For Movement-Based Visual Object Recognitioncited by 5
2022OmniPrint: A Configurable Printed Character Synthesizercited by 5

50 most cited of 137 All 137 (JSON)