mlsanityData to 7 Oct 2026

Retraining with Predicted Hard Labels Provably Increases Model Accuracy

arXiv 2406.11206 ↗First posted 17 Jun 2024cs.LG cs.CR stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

The performance of a model trained with noisy labels is often improved by simply textitretraining the model with its textitown predicted hard labels (i.e., 1/0 labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a linearly separable binary classification setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels. To the best of our knowledge, this is the first such theoretical result. Retraining finds application in improving training with local label differential privacy (DP) which involves training with noisy labels. We empirically show that retraining selectively on the samples for which the predicted label matches the given label significantly improves label DP training at no extra privacy cost; we call this consensus-based retraining. As an example, when training ResNet-18 on CIFAR-100 with ε=3 label DP, we obtain more than 6% improvement in accuracy with consensus-based retraining.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

36 of 53 references.

References · 36

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  6. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  7. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  8. 2018Born Again Neural Networkscited by 539
  9. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  10. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  11. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  12. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
24 more
  1. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  2. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  3. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  4. 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
  5. 2017Learning from Noisy Labels with Distillationcited by 160
  6. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  7. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 148
  8. 2019Deep Self-Learning From Noisy Labelscited by 122
  9. 2021Deep Learning with Label Differential Privacycited by 113
  10. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
  11. 2022Self-Training: A Surveycited by 85
  12. 2020Error-Bounded Correction of Noisy Labelscited by 74
  13. 2020Self-training Avoids Using Spurious Features Under Domain Shiftcited by 69
  14. 2014Private Learning and Sanitization: Pure vs. Approximate Differential Privacycited by 62
  15. 2021Antipodes of Label Differential Privacy: PATE and ALIBIcited by 50
  16. 2019An Introductory Guide to Fano's Inequality with Applications in Statistical Estimationcited by 30
  17. 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 27
  18. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  19. 2022How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysiscited by 18
  20. 2022Regression with Label Differential Privacycited by 14
  21. 2020Statistical and Algorithmic Insights for Semi-supervised Learning with Self-trainingcited by 11
  22. 2022PARS: Pseudo-Label Aware Robust Sample Selection for Learning with Noisy Labelscited by 7
  23. 2023Optimal Unbiased Randomizers for Regression with Label Differential Privacycited by 6
  24. 2023Binary Classification with Instance and Label Dependent Label Noisecited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  2. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  3. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  4. 2025Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passingcited by 0