# Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

arXiv: https://arxiv.org/abs/1910.13427 · HTML: https://mlsanity.com/paper/1910.13427/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 30 of 43)

- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](https://mlsanity.com/paper/2205.07246/) (2022)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [What makes unlearning hard and what to do about it](https://mlsanity.com/paper/2406.01257/) (2024)
- [Can Neural Network Memorization Be Localized?](https://mlsanity.com/paper/2307.09542/) (2023)
- [T-MARS: Improving Visual Representations by Circumventing Text Feature Learning](https://mlsanity.com/paper/2307.03132/) (2023)
- [Characterizing Datapoints via Second-Split Forgetting](https://mlsanity.com/paper/2210.15031/) (2022)
- [Unsupervised Selective Labeling for More Effective Semi-Supervised Learning](https://mlsanity.com/paper/2110.03006/) (2021)
- [Memorization Through the Lens of Curvature of Loss Function Around Samples](https://mlsanity.com/paper/2307.05831/) (2023)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [Why is SAM Robust to Label Noise?](https://mlsanity.com/paper/2405.03676/) (2024)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [Late Stopping: Avoiding Confidently Learning from Mislabeled Examples](https://mlsanity.com/paper/2308.13862/) (2023)
- [Unlocking Accuracy and Fairness in Differentially Private Image Classification](https://mlsanity.com/paper/2308.10888/) (2023)
- [Memorization in deep learning: A survey](https://mlsanity.com/paper/2406.03880/) (2024)
- [The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasets](https://mlsanity.com/paper/2312.05114/) (2023)
- [Dataset Difficulty and the Role of Inductive Bias](https://mlsanity.com/paper/2401.01867/) (2024)
- [Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI](https://mlsanity.com/paper/2403.04551/) (2024)
- [Unveiling Privacy, Memorization, and Input Curvature Links](https://mlsanity.com/paper/2402.18726/) (2024)
- [Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts](https://mlsanity.com/paper/2302.02931/) (2023)
- [SelfMix: Robust Learning Against Textual Label Noise with Self-Mixup Training](https://mlsanity.com/paper/2210.04525/) (2022)
- [Adaptive Conformal Regression with Jackknife+ Rescaled Scores](https://mlsanity.com/paper/2305.19901/) (2023)
- [FUNU: Boosting Machine Unlearning Efficiency by Filtering Unnecessary Unlearning](https://mlsanity.com/paper/2501.16614/) (2025)
- [Minimum Topology Attacks for Graph Neural Networks](https://mlsanity.com/paper/2403.02723/) (2024)
- [Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond](https://mlsanity.com/paper/2503.07501/) (2025)
- [Influence Scores at Scale for Efficient Language Data Sampling](https://mlsanity.com/paper/2311.16298/) (2023)
- [Scalability of memorization-based machine unlearning](https://mlsanity.com/paper/2410.16516/) (2024)
- [Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision](https://mlsanity.com/paper/2310.20153/) (2023)
- [Towards the Memorization Effect of Neural Networks in Adversarial Training](https://mlsanity.com/paper/2106.04794/) (2021)

## Abstract (arXiv)

We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution. We evaluate five methods to score examples in a dataset by how well-represented the examples are, for different plausible definitions of "well-represented", and apply these to four common datasets: MNIST, Fashion-MNIST, CIFAR-10, and ImageNet. Despite being independent approaches, we find all five are highly correlated, suggesting that the notion of being well-represented can be quantified. Among other uses, we find these methods can be combined to identify (a) prototypical examples (that match human expectations); (b) memorized training examples; and, (c) uncommon submodes of the dataset. Further, we show how we can utilize our metrics to determine an improved ordering for curriculum learning, and impact adversarial robustness. We release all metric values on training and test sets we studied.