mlsanityData to 7 Oct 2026

Characterizing Structural Regularities of Labeled Data in Overparameterized Models

arXiv 2002.03206 ↗First posted 8 Feb 2020cs.LG stat.MLcited by 93

Not on the map. Why?

Abstract (arXiv)

Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We show examples of potential applications to the analysis of deep-learning systems.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 17; 2023: 20; 2024: 20; 2025: 20; 2026: 7.

93 papers.

Sort:

By area:

  1. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  2. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  3. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  4. 2022Dataset Condensation with Contrastive Signalscited by 105
  5. 2024What makes unlearning hard and what to do about itcited by 89
  6. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  7. 2023Can Neural Network Memorization Be Localized?cited by 66
  8. 2021Meta-Learning to Compositionally Generalizecited by 62
  9. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60
  10. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  11. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  12. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
38 more
  1. 2023Data Valuation Without Training of a Modelcited by 35
  2. 2022Contrastive Learning with Boosted Memorizationcited by 26
  3. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  4. 2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 24
  5. 2025Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertaintycited by 22
  6. 2022Better Supervisory Signals by Observing Learning Pathscited by 21
  7. 2024Why is SAM Robust to Label Noise?cited by 21
  8. 2022Selective Prediction via Training Dynamicscited by 20
  9. 2022Individual Privacy Accounting for Differentially Private Stochastic Gradient Descentcited by 20
  10. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 20
  11. 2021A Tale Of Two Long Tailscited by 19
  12. 2023Late Stopping: Avoiding Confidently Learning from Mislabeled Examplescited by 17
  13. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  14. 2023Learning Sample Difficulty from Pre-trained Models for Reliable Predictioncited by 15
  15. 2024Memorization in deep learning: A surveycited by 14
  16. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  17. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  18. 2025From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularizationcited by 12
  19. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11
  20. 2023A Bag-of-Prototypes Representation for Dataset-Level Applicationscited by 11
  21. 2023Sample based Explanations via Generalized Representerscited by 10
  22. 2022Memory Population in Continual Learning via Outlier Eliminationcited by 9
  23. 2024Unveiling Privacy, Memorization, and Input Curvature Linkscited by 9
  24. 2024Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Taskscited by 9
  25. 2023SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storagecited by 8
  26. 2024Predicting the Susceptibility of Examples to Catastrophic Forgettingcited by 7
  27. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  28. 2025Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted datacited by 7
  29. 2021Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Losscited by 6
  30. 2022Predicting on the Edge: Identifying Where a Larger Model Does Bettercited by 6
  31. 2022Angular Gap: Reducing the Uncertainty of Image Difficulty through Model Calibrationcited by 6
  32. 2023Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Developmentcited by 6
  33. 2023Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Modelscited by 6
  34. 2023What do larger image classifiers memorise?cited by 6
  35. 2024lpNTK: Better Generalisation with Less Data via Sample Interaction During Learningcited by 6
  36. 2024A Large-Scale Study on Video Action Dataset Condensationcited by 6
  37. 2022Measuring Unintended Memorisation of Unique Private Features in Neural Networkscited by 5
  38. 2022Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficultycited by 5

50 most cited of 93 All 93 (JSON)