mlsanityData to 7 Oct 2026

On Tilted Losses in Machine Learning: Theory and Applications

arXiv 2109.06141 ↗First posted 13 Sep 2021cs.LG cs.IT math.ITcited by 42

Not on the map. Why?

Abstract (arXiv)

Exponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. Despite its prevalence in related fields, tilting has not seen widespread use in machine learning. In this work, we aim to bridge this gap by exploring the use of tilting in risk minimization. We study a simple extension to ERM -- tilted empirical risk minimization (TERM) -- which uses exponential tilting to flexibly tune the impact of individual losses. The resulting framework has several useful properties: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to the tail probability of losses. Our work makes rigorous connections between TERM and related objectives, such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization (DRO). We develop batch and stochastic first-order optimization methods for solving TERM, provide convergence guarantees for the solvers, and show that the framework can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. Despite the straightforward modification TERM makes to traditional ERM objectives, we find that the framework can consistently outperform ERM and deliver competitive performance with state-of-the-art, problem-specific approaches.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 64 of its 142 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

64 of 142 references.

References · 64

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  8. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  9. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  10. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  11. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  12. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
38 more
  1. 2019Agnostic Federated Learningcited by 451
  2. 2019Fair Resource Allocation in Federated Learningcited by 389
  3. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 368
  4. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  5. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  6. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  7. 2017How to Escape Saddle Points Efficientlycited by 284
  8. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  10. 2018Empirical Risk Minimization under Fairness Constraintscited by 226
  11. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  12. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  13. 2016Variance-based regularization with convex objectivescited by 153
  14. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  15. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  16. 2018Sever: A Robust Meta-Algorithm for Stochastic Optimizationcited by 123
  17. 2017Learning From Noisy Large-Scale Datasets With Minimal Supervisioncited by 115
  18. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  19. 2020Tilted Empirical Risk Minimizationcited by 103
  20. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  21. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 93
  22. 2018The Price of Fair PCA: One Extra Dimensioncited by 67
  23. 2018A Single Time-Scale Stochastic Approximation Method for Nested Stochastic Optimizationcited by 53
  24. 2017Learning with Average Top-k Losscited by 52
  25. 2020FR-Train: A Mutual Information-Based Approach to Fair and Robust Trainingcited by 52
  26. 2020Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularizationcited by 49
  27. 2016Accelerating Stochastic Composition Optimizationcited by 46
  28. 2021Distributionally Robust Learningcited by 44
  29. 2015Robust Regression via Hard Thresholdingcited by 39
  30. 2019Adaptive Sampling for Stochastic Risk-Averse Learningcited by 39
  31. 2020Efficient Search of First-Order Nash Equilibria in Nonconvex-Concave Smooth Min-Max Problemscited by 39
  32. 2019Toward a better trade-off between performance and fairness with kernel-based distribution matchingcited by 36
  33. 2019R'enyi Fair Inferencecited by 33
  34. 2020Active Sampling for Min-Max Fairnesscited by 33
  35. 2017Learning Deep Networks from Noisy Labels with Dropout Regularizationcited by 32
  36. 2017Biased Importance Sampling for Deep Neural Network Trainingcited by 31
  37. 2018PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Reviewcited by 30
  38. 2019Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspectivecited by 26

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 3; 2024: 16; 2025: 6; 2026: 6.

42 papers.

Sort:

By area:

  1. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  2. 2024Asymptotics of Language Model Alignmentcited by 46
  3. 2022Probabilistically Robust Learning: Balancing Average- and Worst-case Performancecited by 36
  4. 2024InfAlign: Inference-aware language model alignmentcited by 34
  5. 2022Rank-based Decomposable Losses in Machine Learning: A Surveycited by 26
  6. 2022Proportional Fairness in Federated Learningcited by 20
  7. 2022Optimal Algorithms for Stochastic Multi-Level Compositional Optimizationcited by 17
  8. 2022Stochastic Constrained DRO with a Complexity Independent of Sample Sizecited by 16
  9. 2024Domain Generalisation via Imprecise Learningcited by 14
  10. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  11. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  12. 2024Distributionally Robust Performative Predictioncited by 7
30 more
  1. 2023Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimizationcited by 6
  2. 2024Tilted Sharpness-Aware Minimizationcited by 6
  3. 2022On Proper Learnability between Average- and Worst-case Robustnesscited by 5
  4. 2021A Survey of Learning Criteria Going Beyond the Usual Riskcited by 4
  5. 2022(Im)possibility of Collective Intelligencecited by 4
  6. 2024Generalization and Robustness of the Tilted Empirical Riskcited by 4
  7. 2022Flexible risk design using bi-directional dispersioncited by 3
  8. 2024Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergencecited by 3
  9. 2022Outlier-Robust Group Inference via Gradient Space Clusteringcited by 2
  10. 2023Improving Identity-Robustness for Face Modelscited by 2
  11. 2024Projection-Free Variance Reduction Methods for Stochastic Constrained Multi-Level Compositional Optimizationcited by 2
  12. 2024Stability Evaluation via Distributional Perturbation Analysiscited by 1
  13. 2024On the KL-Divergence-based Robust Satisficing Modelcited by 1
  14. 2025Enhancing Diffusion Model Guidance through Calibration and Regularizationcited by 1
  15. 2025Power Homotopy for Zeroth-Order Non-Convex Optimizationscited by 1
  16. 2026Toward Calibrated Mixture-of-Experts Under Distribution Shiftcited by 1
  17. 2023Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspectivecited by 0
  18. 2024Criterion Collapse and Loss Distribution Controlcited by 0
  19. 2024Efficient k-means with Individual Fairness via Exponential Tiltingcited by 0
  20. 2024Exponential tilting of subweibull distributionscited by 0
  21. 2024On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairnesscited by 0
  22. 2024Making Robust Generalizers Less Rigid with Loss Concentrationcited by 0
  23. 2025Data Heterogeneity Modeling for Trustworthy Machine Learningcited by 0
  24. 2025Benefits of Online Tilted Empirical Risk Minimization: A Case Study of Outlier Detection and Robust Regressioncited by 0
  25. 2025Zeroth-Order Sharpness-Aware Learning with Exponential Tiltingcited by 0
  26. 2026Progressive Power Homotopy for Non-convex Optimizationcited by 0
  27. 2026Learnable Chernoff Baselines for Inference-Time Alignmentcited by 0
  28. 2026DuetMoE: Coupling Inter- and Intra-Subgroup Robustness for Fair Medical Image Analysiscited by 0
  29. 2026FairMean: Promoting Fairness in Distributed Learning under Label Poisoning Attackscited by 0
  30. 2026Optimal Momentum Methods for Stochastic Multilevel Compositional Optimizationcited by 0