mlsanityData to 14 Oct 2026

Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

arXiv 1910.13427 ↗First posted 29 Oct 2019cs.LG stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution. We evaluate five methods to score examples in a dataset by how well-represented the examples are, for different plausible definitions of "well-represented", and apply these to four common datasets: MNIST, Fashion-MNIST, CIFAR-10, and ImageNet. Despite being independent approaches, we find all five are highly correlated, suggesting that the notion of being well-represented can be quantified. Among other uses, we find these methods can be combined to identify (a) prototypical examples (that match human expectations); (b) memorized training examples; and, (c) uncommon submodes of the dataset. Further, we show how we can utilize our metrics to determine an improved ordering for curriculum learning, and impact adversarial robustness. We release all metric values on training and test sets we studied.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 14; 2024: 9; 2025: 5; 2026: 3.

43 papers.

Sort:

By area:

  1. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  2. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  3. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  4. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
  5. 2024What makes unlearning hard and what to do about itcited by 89
  6. 2023Can Neural Network Memorization Be Localized?cited by 69
  7. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  8. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  9. 2021Unsupervised Selective Labeling for More Effective Semi-Supervised Learningcited by 27
  10. 2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 27
  11. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  12. 2024Why is SAM Robust to Label Noise?cited by 21
31 more
  1. 2021A Tale Of Two Long Tailscited by 19
  2. 2023Late Stopping: Avoiding Confidently Learning from Mislabeled Examplescited by 17
  3. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  4. 2024Memorization in deep learning: A surveycited by 15
  5. 2023The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasetscited by 14
  6. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  7. 2024Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AIcited by 11
  8. 2024Unveiling Privacy, Memorization, and Input Curvature Linkscited by 10
  9. 2023Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shiftscited by 9
  10. 2022SelfMix: Robust Learning Against Textual Label Noise with Self-Mixup Trainingcited by 8
  11. 2023Adaptive Conformal Regression with Jackknife+ Rescaled Scorescited by 8
  12. 2025FUNU: Boosting Machine Unlearning Efficiency by Filtering Unnecessary Unlearningcited by 7
  13. 2024Minimum Topology Attacks for Graph Neural Networkscited by 6
  14. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  15. 2023Influence Scores at Scale for Efficient Language Data Samplingcited by 5
  16. 2024Scalability of memorization-based machine unlearningcited by 5
  17. 2023Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervisioncited by 4
  18. 2021Towards the Memorization Effect of Neural Networks in Adversarial Trainingcited by 3
  19. 2022Impact of Strategic Sampling and Supervision Policies on Semi-supervised Learningcited by 2
  20. 2023Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applicationscited by 2
  21. 2023Read, Look or Listen? What's Needed for Solving a Multimodal Datasetcited by 2
  22. 2023SoK: Memorisation in machine learningcited by 1
  23. 2024Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attackscited by 1
  24. 2025FairDropout: Using Example-Tied Dropout to Enhance Generalization of Minority Groupscited by 1
  25. 2025The Easy Path to Robustness: Coreset Selection using Sample Hardnesscited by 1
  26. 2026Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Datacited by 1
  27. 2022Tensions Between the Proxies of Human Values in AIcited by 0
  28. 2023Extended Linear Regression: A Kalman Filter Approach for Minimizing Loss via Area Under the Curvecited by 0
  29. 2025Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Oddscited by 0
  30. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0
  31. 2026Benchmarking Unlearning for Vision Transformerscited by 0