mlsanityData to 7 Oct 2026

Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity

arXiv 2112.14869 ↗First posted 30 Dec 2021cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification that are formulated from distributionally robust optimization (DRO) perspective, where the uncertainty in the given label information are modeled and captured by taking the worse case of distributional weights. The benefits of this perspective are several fold: (i) it provides a unified framework to explain the classical cross-entropy (CE) loss and SVM loss and their variants, (ii) it includes a special family corresponding to the temperature-scaled CE loss, which is widely adopted but poorly understood; (iii) it allows us to achieve adaptivity to the uncertainty degree of label information at an instance level. Our contributions include: (1) we study both consistency and robustness by establishing top-k (forall k≥ 1) consistency of LDR losses for multi-class classification, and a negative result that a top-1 consistent and symmetric robust loss cannot achieve top-k consistency simultaneously for all k≥ 2; (2) we propose a new adaptive LDR loss that automatically adapts the individualized temperature parameter to the noise degree of class label of each instance; (3) we demonstrate stable and competitive performance for the proposed adaptive LDR loss on 7 benchmark datasets under 6 noisy label and 1 clean settings against 13 loss functions, and on one real-world noisy dataset. The code is open-sourced at urlhttps://github.com/Optimization-AI/ICML2023LDR.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

33 of 49 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  7. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  8. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  9. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  10. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  11. 2018Deep Learning for Classical Japanese Literaturecited by 415
  12. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
21 more
  1. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  2. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  3. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  4. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
  5. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  6. 2016Variance-based regularization with convex objectivescited by 153
  7. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  8. 2018Masking: A New Perspective of Noisy Supervisioncited by 93
  9. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  10. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  11. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  12. 2017On Loss Functions for Deep Neural Networks in Classificationcited by 68
  13. 2021Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularizationcited by 62
  14. 2018Universal Language Model Fine-tuning for Text Classificationcited by 50
  15. 2021Asymmetric Loss Functions for Learning with Noisy Labelscited by 46
  16. 2015Loss Functions for Top-k Error: Analysis and Insightscited by 38
  17. 2019On the Consistency of Top-k Surrogate Lossescited by 35
  18. 2016Minimizing the Maximal Loss: How and Why?cited by 30
  19. 2015Top-k Multiclass SVMcited by 28
  20. 2016Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classificationcited by 20
  21. 2021Memorization in Deep Neural Networks: Does the Loss Function matter?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 3; 2026: 1.

11 papers.

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
  2. 2024To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DROcited by 11
  3. 2025Loss Functions and Operators Generated by f-Divergencescited by 10
  4. 2023Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimizationcited by 8
  5. 2025ε-Softmax: Approximating One-Hot Vectors for Mitigating Label Noisecited by 6
  6. 2024Gradient Aligned Regression via Pairwise Lossescited by 1
  7. 2026Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labelscited by 1
  8. 2023Scalable Label Distribution Learning for Multi-Label Classificationcited by 0
  9. 2024Wasserstein Distributionally Robust Multiclass Support Vector Machinecited by 0
  10. 2024Group & Reweight: A Novel Cost-Sensitive Approach to Mitigating Class Imbalance in Network Traffic Classificationcited by 0
  11. 2025Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labelscited by 0