mlsanityData to 7 Oct 2026

Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmark

arXiv 2410.15772 ↗First posted 21 Oct 2024cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Mislabeled examples are ubiquitous in real-world machine learning datasets, advocating the development of techniques for automatic detection. We show that most mislabeled detection methods can be viewed as probing trained machine learning models using a few core principles. We formalize a modular framework that encompasses these methods, parameterized by only 4 building blocks, as well as a Python library that demonstrates that these principles can actually be implemented. The focus is on classifier-agnostic concepts, with an emphasis on adapting methods developed for deep learning models to non-deep classifiers for tabular data. We benchmark existing methods on (artificial) Completely At Random (NCAR) as well as (realistic) Not At Random (NNAR) labeling noise from a variety of tasks with imperfect labeling rules. This benchmark provides new insights as well as limitations of existing methods in this setup.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 60 of its 112 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

60 of 112 references.

References · 60

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2017A Closer Look at Memorization in Deep Networkscited by 936
  5. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  6. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  8. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  9. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  10. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  11. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
38 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  2. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  3. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  4. 2020Learning under Concept Drift: A Reviewcited by 470
  5. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  6. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  7. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  8. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  9. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  10. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  11. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  12. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  13. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  14. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  15. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  16. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
  17. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  18. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  19. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  20. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  21. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  22. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  23. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 121
  24. 2020Estimating Example Difficulty Using Variance of Gradientscited by 96
  25. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  26. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  27. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 57
  28. 2020Deep k-NN for Noisy Labelscited by 55
  29. 2020Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languagescited by 52
  30. 2022ProMix: Combating Label Noise via Maximizing Clean Sample Utilitycited by 48
  31. 2021LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environmentcited by 40
  32. 2022Identifiability of Label Noise Transition Matrixcited by 34
  33. 2021End-to-End Weak Supervisioncited by 33
  34. 2021Estimating Instance-dependent Bayes-label Transition Matrix using a Deep Neural Networkcited by 30
  35. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  36. 2022Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular datacited by 22
  37. 2024Conformalized Credal Set Predictorscited by 21
  38. 2021A Tale Of Two Long Tailscited by 19

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Calibration improves detection of mislabeled examplescited by 0