# On Tilted Losses in Machine Learning: Theory and Applications

arXiv: https://arxiv.org/abs/2109.06141 · HTML: https://mlsanity.com/paper/2109.06141/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 142 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 30 of 64)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://mlsanity.com/paper/1503.02101/) (2015)
- [Empirical Risk Minimization under Fairness Constraints](https://mlsanity.com/paper/1802.08626/) (2018)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Large-Scale Methods for Distributionally Robust Optimization](https://mlsanity.com/paper/2010.05893/) (2020)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods](https://mlsanity.com/paper/1902.08297/) (2019)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)
- [Sever: A Robust Meta-Algorithm for Stochastic Optimization](https://mlsanity.com/paper/1803.02815/) (2018)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [Tilted Empirical Risk Minimization](https://mlsanity.com/paper/2007.01162/) (2020)

## Built on by (most cited first, 30 of 43)

- [Theoretical guarantees on the best-of-n alignment policy](https://mlsanity.com/paper/2401.01879/) (2024)
- [Asymptotics of Language Model Alignment](https://mlsanity.com/paper/2404.01730/) (2024)
- [Probabilistically Robust Learning: Balancing Average- and Worst-case Performance](https://mlsanity.com/paper/2202.01136/) (2022)
- [InfAlign: Inference-aware language model alignment](https://mlsanity.com/paper/2412.19792/) (2024)
- [Rank-based Decomposable Losses in Machine Learning: A Survey](https://mlsanity.com/paper/2207.08768/) (2022)
- [Proportional Fairness in Federated Learning](https://mlsanity.com/paper/2202.01666/) (2022)
- [Optimal Algorithms for Stochastic Multi-Level Compositional Optimization](https://mlsanity.com/paper/2202.07530/) (2022)
- [Stochastic Constrained DRO with a Complexity Independent of Sample Size](https://mlsanity.com/paper/2210.05740/) (2022)
- [Domain Generalisation via Imprecise Learning](https://mlsanity.com/paper/2404.04669/) (2024)
- [Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs](https://mlsanity.com/paper/2510.17472/) (2025)
- [Understanding new tasks through the lens of training data via exponential tilting](https://mlsanity.com/paper/2205.13577/) (2022)
- [Distributionally Robust Performative Prediction](https://mlsanity.com/paper/2412.04346/) (2024)
- [Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization](https://mlsanity.com/paper/2306.09222/) (2023)
- [Tilted Sharpness-Aware Minimization](https://mlsanity.com/paper/2410.22656/) (2024)
- [On Proper Learnability between Average- and Worst-case Robustness](https://mlsanity.com/paper/2211.05656/) (2022)
- [A Survey of Learning Criteria Going Beyond the Usual Risk](https://mlsanity.com/paper/2110.04996/) (2021)
- [(Im)possibility of Collective Intelligence](https://mlsanity.com/paper/2206.02786/) (2022)
- [Generalization and Robustness of the Tilted Empirical Risk](https://mlsanity.com/paper/2409.19431/) (2024)
- [Flexible risk design using bi-directional dispersion](https://mlsanity.com/paper/2203.14434/) (2022)
- [Outlier-Robust Group Inference via Gradient Space Clustering](https://mlsanity.com/paper/2210.06759/) (2022)
- [Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergence](https://mlsanity.com/paper/2405.00454/) (2024)
- [Improving Identity-Robustness for Face Models](https://mlsanity.com/paper/2304.03838/) (2023)
- [Projection-Free Variance Reduction Methods for Stochastic Constrained Multi-Level Compositional Optimization](https://mlsanity.com/paper/2406.03787/) (2024)
- [Stability Evaluation via Distributional Perturbation Analysis](https://mlsanity.com/paper/2405.03198/) (2024)
- [On the KL-Divergence-based Robust Satisficing Model](https://mlsanity.com/paper/2408.09157/) (2024)
- [Enhancing Diffusion Model Guidance through Calibration and Regularization](https://mlsanity.com/paper/2511.05844/) (2025)
- [Power Homotopy for Zeroth-Order Non-Convex Optimizations](https://mlsanity.com/paper/2511.13592/) (2025)
- [Toward Calibrated Mixture-of-Experts Under Distribution Shift](https://mlsanity.com/paper/2606.20544/) (2026)
- [Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspective](https://mlsanity.com/paper/2311.01047/) (2023)
- [Criterion Collapse and Loss Distribution Control](https://mlsanity.com/paper/2402.09802/) (2024)

## Abstract (arXiv)

Exponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. Despite its prevalence in related fields, tilting has not seen widespread use in machine learning. In this work, we aim to bridge this gap by exploring the use of tilting in risk minimization. We study a simple extension to ERM -- tilted empirical risk minimization (TERM) -- which uses exponential tilting to flexibly tune the impact of individual losses. The resulting framework has several useful properties: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to the tail probability of losses. Our work makes rigorous connections between TERM and related objectives, such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization (DRO). We develop batch and stochastic first-order optimization methods for solving TERM, provide convergence guarantees for the solvers, and show that the framework can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. Despite the straightforward modification TERM makes to traditional ERM objectives, we find that the framework can consistently outperform ERM and deliver competitive performance with state-of-the-art, problem-specific approaches.