mlsanityData to 7 Oct 2026

A Curriculum View of Robust Loss Functions

arXiv 2305.02139 ↗First posted 3 May 2023cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Robust loss functions are designed to combat the adverse impacts of label noise, whose robustness is typically supported by theoretical bounds agnostic to the training dynamics. However, these bounds may fail to characterize the empirical performance as it remains unclear why robust loss functions can underfit. We show that most loss functions can be rewritten into a form with the same class-score margin and different sample-weighting functions. The resulting curriculum view provides a straightforward analysis of the training dynamics, which helps attribute underfitting to diminished average sample weights and noise robustness to larger weights for clean samples. We show that simple fixes to the curriculums can make underfitting robust loss functions competitive with the state-of-the-art, and training schedules can substantially affect the noise robustness even with robust loss functions. Code is available at urlgithub.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

35 of 57 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  9. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  10. 2017A Closer Look at Memorization in Deep Networkscited by 936
  11. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  12. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
23 more
  1. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  2. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  3. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  4. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  5. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  6. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  7. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  8. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  9. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  10. 2020Does label smoothing mitigate label noise?cited by 204
  11. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  12. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  13. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  14. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  15. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  16. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  17. 2018An Empirical Exploration of Curriculum Learning for Neural Machine Translationcited by 58
  18. 2021Asymmetric Loss Functions for Learning with Noisy Labelscited by 46
  19. 2019How does Early Stopping Help Generalization against Label Noise?cited by 41
  20. 2019IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matterscited by 33
  21. 2021Noisy-Labeled NER with Confidence Estimationcited by 28
  22. 2022On the Generalization Mystery in Deep Learningcited by 25
  23. 2019Searching to Exploit Memorization Effect in Learning from Corrupted Labelscited by 11

Built on by

Not cited yet.