mlsanityData to 7 Oct 2026

Training Over-parameterized Models with Non-decomposable Objectives

arXiv 2107.04641 ↗First posted 9 Jul 2021cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints. Popular techniques for optimizing such non-decomposable objectives reduce the problem into a sequence of cost-sensitive learning tasks, each of which is then solved by re-weighting the training loss with example-specific costs. We point out that the standard approach of re-weighting the loss to incorporate label costs can produce unsatisfactory results when used to train over-parameterized models. As a remedy, we propose new cost-sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model. Through experiments on benchmark image datasets, we showcase the effectiveness of our approach in training ResNet models with common robust and constrained optimization objectives.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 48 of its 113 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

48 of 113 references.

References · 48

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  6. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  7. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  8. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  10. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  11. 2020Long-tail learning via logit adjustmentcited by 605
  12. 2018A Reductions Approach to Fair Classificationcited by 585
36 more
  1. 2018Born Again Neural Networkscited by 539
  2. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  3. 2019Agnostic Federated Learningcited by 451
  4. 2015Policy Distillationcited by 414
  5. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  6. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  7. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  8. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  9. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  10. 2015Cost Sensitive Learning of Deep Feature Representations from Imbalanced Datacited by 167
  11. 2020Seesaw Loss for Long-Tailed Instance Segmentationcited by 146
  12. 2020Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspectivecited by 143
  13. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  14. 2020Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasetscited by 122
  15. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  16. 2017The Devil is in the Tails: Fine-grained Classification in the Wildcited by 89
  17. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  18. 2016Satisfying Real-world Goals with Dataset Constraintscited by 68
  19. 2018Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraintscited by 53
  20. 2016Scalable Learning of Non-Decomposable Objectivescited by 40
  21. 2017Robust Optimization for Non-Convex Objectivescited by 34
  22. 2019Addressing the Loss-Metric Mismatch with Adaptive Loss Alignmentcited by 33
  23. 2021PML: Progressive Margin Loss for Long-tailed Age Classificationcited by 32
  24. 2015Training Deep Neural Networks via Direct Loss Minimizationcited by 25
  25. 2016Reviving Threshold-Moving: a Simple Plug-in Bagging Ensemble for Binary and Multiclass Imbalanced Datacited by 23
  26. 2021Balanced Knowledge Distillation for Long-tailed Learningcited by 23
  27. 2014Convex Calibration Dimension for Multiclass Loss Matricescited by 21
  28. 2014Online and Stochastic Gradient Methods for Non-decomposable Loss Functionscited by 21
  29. 2019To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributionscited by 20
  30. 2015Optimizing Non-decomposable Performance Measures: A Tale of Two Classescited by 17
  31. 2018Binary Classification with Karmic, Threshold-Quasi-Concave Metricscited by 11
  32. 2018Optimizing Non-decomposable Measures with Deep Networkscited by 10
  33. 2019AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learningcited by 8
  34. 2016Online Optimization Methods for the Quantification Problemcited by 4
  35. 2021A Minimax Probability Machine for Non-Decomposable Performance Measurescited by 4
  36. 2018tau-FPL: Tolerance-Constrained Learning in Linear Timecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 1; 2025: 2; 2026: 2.

11 papers.

  1. 2022Rank-based Decomposable Losses in Machine Learning: A Surveycited by 26
  2. 2023Long-Tailed Partial Label Learning via Dynamic Rebalancingcited by 22
  3. 2022Robust Distillation for Worst-class Performancecited by 6
  4. 2022Importance Tempering: Group Robustness for Overparameterized Modelscited by 6
  5. 2023Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metricscited by 4
  6. 2023A comprehensive theoretical framework for the optimization of neural networks classification performance with respect to weighted metricscited by 3
  7. 2024Selective Mixup Fine-Tuning for Optimizing Non-Decomposable Objectivescited by 2
  8. 2026DRAGON: Robust Classification for Very Large Collections of Software Repositoriescited by 1
  9. 2025Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentationcited by 0
  10. 2025FedFACT: A Provable Framework for Controllable Group-Fairness Calibration in Federated Learningcited by 0
  11. 2026Demystifying the Optimal Fair Classifier in Multi-Class Classificationcited by 0