mlsanityData to 7 Oct 2026

A Tale Of Two Long Tails

arXiv 2107.13098 ↗First posted 27 Jul 2021cs.CV cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

As machine learning models are increasingly employed to assist human decision-makers, it becomes critical to communicate the uncertainty associated with these model predictions. However, the majority of work on uncertainty has focused on traditional probabilistic or ranking approaches - where the model assigns low probabilities or scores to uncertain examples. While this captures what examples are challenging for the model, it does not capture the underlying source of the uncertainty. In this work, we seek to identify examples the model is uncertain about and characterize the source of said uncertainty. We explore the benefits of designing a targeted intervention - targeted data augmentation of the examples where the model is uncertain over the course of training. We investigate whether the rate of learning in the presence of additional information differs between atypical and noisy examples? Our results show that this is indeed the case, suggesting that well-designed interventions over the course of training can be an effective way to characterize and distinguish between different sources of uncertainty.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

43 of 69 references.

References · 43

  1. 2016Wide Residual Networkscited by 3,057
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  8. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  9. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  12. 2020Are we done with ImageNet?cited by 329
31 more
  1. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  2. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  3. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  4. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  5. 2019Probabilistic End-to-end Noise Correction for Learning with Noisy Labelscited by 210
  6. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  7. 2020Characterising Bias in Compressed Modelscited by 162
  8. 2018Learning to Learn from Noisy Labeled Datacited by 160
  9. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  10. 2019A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysiscited by 150
  11. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  12. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  13. 2021The Benchmark Lotterycited by 133
  14. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  15. 2020Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspectivecited by 120
  16. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  17. 2020Estimating Example Difficulty Using Variance of Gradientscited by 96
  18. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  19. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  20. 2019Combating Label Noise in Deep Learning Using Abstentioncited by 86
  21. 2016Coresets for Scalable Bayesian Logistic Regressioncited by 79
  22. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  23. 2015The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classificationcited by 76
  24. 2020A Topological Filter for Learning with Label Noisecited by 76
  25. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  26. 2020A Survey of Active Learning for Text Classification using Deep Neural Networkscited by 50
  27. 2019Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applicationscited by 43
  28. 2020A Study of Gradient Variance in Deep Learningcited by 27
  29. 2021Core-set Sampling for Efficient Neural Architecture Searchcited by 23
  30. 2021Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curationcited by 13
  31. 2021When does loss-based prioritization fail?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 5; 2024: 1; 2025: 2; 2026: 1.

19 papers.

Sort:

  1. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  2. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  3. 2022Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergycited by 20
  4. 2022A Theoretical Analysis of the Learning Dynamics under Class Imbalancecited by 20
  5. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18
  6. 2022CrossSplit: Mitigating Label Noise Memorization through Data Splittingcited by 12
  7. 2022Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Treescited by 9
  8. 2022Robust Distillation for Worst-class Performancecited by 6
  9. 2023Data Optimization in Deep Learning: A Surveycited by 6
  10. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  11. 2022When less is more: Simplifying inputs aids neural network understandingcited by 4
  12. 2022Exploring the Learning Difficulty of Data Theory and Measurecited by 4
7 more
  1. 2023Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translationcited by 4
  2. 2025The Impact of Coreset Selection on Spurious Correlations and Group Robustnesscited by 4
  3. 2023Designing Data: Proactive Data Collection and Iteration for Machine Learningcited by 2
  4. 2024Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmarkcited by 1
  5. 2026APEX: Probing Neural Networks via Activation Perturbationcited by 1
  6. 2023Towards Estimating Transferability using Hard Subsetscited by 0
  7. 2023Padding Aware Neuronscited by 0