mlsanityData to 7 Oct 2026

Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI

arXiv 2403.04551 ↗First posted 7 Mar 2024cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Characterizing samples that are difficult to learn from is crucial to developing highly performant ML models. This has led to numerous Hardness Characterization Methods (HCMs) that aim to identify "hard" samples. However, there is a lack of consensus regarding the definition and evaluation of "hardness". Unfortunately, current HCMs have only been evaluated on specific types of hardness and often only qualitatively or with respect to downstream performance, overlooking the fundamental quantitative identification task. We address this gap by presenting a fine-grained taxonomy of hardness types. Additionally, we propose the Hardness Characterization Analysis Toolkit (H-CAT), which supports comprehensive and quantitative benchmarking of HCMs across the hardness taxonomy and can easily be extended to new HCMs, hardness types, and datasets. We use H-CAT to evaluate 13 different HCMs across 8 hardness types. This comprehensive evaluation encompassing over 14K setups uncovers strengths and weaknesses of different HCMs, leading to practical tips to guide HCM selection and future development. Our findings highlight the need for more comprehensive HCM evaluation, while we hope our hardness taxonomy and toolkit will advance the principled evaluation and uptake of data-centric AI methods.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 39 of its 74 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

39 of 74 references.

References · 39

  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  3. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  4. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  5. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  6. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  7. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  8. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  9. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  10. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  11. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  12. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
27 more
  1. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  2. 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
  3. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 126
  4. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 121
  5. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  6. 2021Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularizationcited by 62
  7. 2019Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applicationscited by 43
  8. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  9. 2021A Data Quality-Driven View of MLOpscited by 31
  10. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  11. 2022The Principles of Data-Centric AI (DCAI)cited by 26
  12. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25
  13. 2023Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmarkcited by 23
  14. 2022Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular datacited by 22
  15. 2022From Noisy Prediction to True Label: Noisy Prediction Calibration via Generative Modelcited by 21
  16. 2021Data Quality Toolkit: Automatic assessment of data quality and remediation for machine learning datasetscited by 18
  17. 2022Transformer-based out-of-distribution detection for clinically safe segmentationcited by 16
  18. 2022Data-SUITE: Data-centric identification of in-distribution incongruous examplescited by 12
  19. 2022Learning from Training Dynamics: Identifying Mislabeled Data Beyond Manually Designed Featurescited by 9
  20. 2021Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Datacited by 7
  21. 2022Raising the Bar on the Evaluation of Out-of-Distribution Detectioncited by 7
  22. 2022Out of Distribution Detection via Neural Network Anchoringcited by 6
  23. 2023TRIAGE: Characterizing and auditing training data for improved regressioncited by 6
  24. 2022What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorizationcited by 5
  25. 2022The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labelscited by 3
  26. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 3
  27. 2022Repairing Systematic Outliers by Learning Clean Subspaces in VAEscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 1.

11 papers.

  1. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  2. 2024Matchmaker: Self-Improving Large Language Model Programs for Schema Matchingcited by 9
  3. 2024Targeted synthetic data generation for tabular data via hardness characterizationcited by 4
  4. 2025The Impact of Coreset Selection on Spurious Correlations and Group Robustnesscited by 4
  5. 2024You can't handle the (dirty) truth: Data-centric insights improve pseudo-labelingcited by 3
  6. 2025Reducing Class Bias In Data-Balanced Datasets Through Hardness-Based Resamplingcited by 0
  7. 2025Draw a Portrait of Your Graph Data: An Instance-Level Profiling Framework for Graph-Structured Datacited by 0
  8. 2025LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Trainingcited by 0
  9. 2025Filtering instances and rejecting predictions to obtain reliable models in healthcarecited by 0
  10. 2025What's the next frontier for Data-centric AI? Data Savvy Agentscited by 0
  11. 2026LiNC: Lightweight Noise Correction via Per-Sample Trust and Gaussian Mixture Modelingcited by 0