mlsanityData to 7 Oct 2026

Human uncertainty makes classification more robust

arXiv 1908.07086 ↗First posted 19 Aug 2019cs.CVcited by 231

Not on the map. Why?

Abstract (arXiv)

The classification performance of deep neural networks has begun to asymptote at near-perfect levels. However, their ability to generalize outside the training set and their robustness to adversarial attacks have not. In this paper, we make progress on this problem by training with full label distributions that reflect human perceptual uncertainty. We first present a new benchmark dataset which we call CIFAR10H, containing a full distribution of human labels for each image of the CIFAR10 test set. We then show that, while contemporary classifiers fail to exhibit human-like uncertainty on their own, explicit training on our dataset closes this gap, supports improved generalization to increasingly out-of-training-distribution test datasets, and confers robustness to adversarial attacks.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 36; 2023: 38; 2024: 46; 2025: 38; 2026: 45.

231 papers.

Sort:

By area:

  1. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  2. 2024Are We Done with MMLU?cited by 178
  3. 2023Getting aligned on representational alignmentcited by 142
  4. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  5. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  6. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  7. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  8. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  9. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  10. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  11. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  12. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
38 more
  1. 2021Active label cleaning for improved dataset quality under resource constraintscited by 44
  2. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  3. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  4. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  5. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  6. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  7. 2023CLIP in Medical Imaging: A Surveycited by 40
  8. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  9. 2021AmbiFC: Fact-Checking Ambiguous Claims with Evidencecited by 38
  10. 2023Improving neural network representations using human similarity judgmentscited by 38
  11. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  12. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  13. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  14. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  15. 2023Conformal prediction under ambiguous ground truthcited by 31
  16. 2022To Aggregate or Not? Learning with Separate Noisy Labelscited by 29
  17. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 28
  18. 2021Human-Aided Saliency Maps Improve Generalization of Deep Learningcited by 27
  19. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  20. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  21. 2022Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classificationcited by 24
  22. 2023Adaptive conformal classification with noisy labelscited by 24
  23. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24
  24. 2021Exploring Segment-level Semantics for Online Phase Recognition from Surgical Videoscited by 23
  25. 2025E-Values Expand the Scope of Conformal Predictioncited by 23
  26. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  27. 2021Capturing the objects of vision with neural networkscited by 21
  28. 2022Better Supervisory Signals by Observing Learning Pathscited by 21
  29. 2024Conformalized Credal Set Predictorscited by 21
  30. 2025Continuous Thought Machinescited by 21
  31. 2025Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)cited by 20
  32. 2021Salient Objects in Cluttercited by 19
  33. 2024Conformalized Credal Regions for Classification with Ambiguous Ground Truthcited by 19
  34. 2024Annotator-Centric Active Learning for Subjective NLP Taskscited by 18
  35. 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
  36. 2023ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variationcited by 17
  37. 2023Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labelscited by 17
  38. 2021I Find Your Lack of Uncertainty in Computer Vision Disturbingcited by 16

50 most cited of 231 All 231 (JSON)