# Making Robust Generalizers Less Rigid with Loss Concentration

arXiv: https://arxiv.org/abs/2408.03619 · HTML: https://mlsanity.com/paper/2408.03619/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 26 of 26)

- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [Tilted Empirical Risk Minimization](https://mlsanity.com/paper/2007.01162/) (2020)
- [Fairness risk measures](https://mlsanity.com/paper/1901.08665/) (2019)
- [High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails](https://mlsanity.com/paper/2106.14343/) (2021)
- [Do We Need Zero Training Loss After Achieving Zero Training Error?](https://mlsanity.com/paper/2002.08709/) (2020)
- [DORO: Distributional and Outlier Robust Optimization](https://mlsanity.com/paper/2106.06142/) (2021)
- [On Tilted Losses in Machine Learning: Theory and Applications](https://mlsanity.com/paper/2109.06141/) (2021)
- [Adaptive Sampling for Stochastic Risk-Averse Learning](https://mlsanity.com/paper/1910.12511/) (2019)
- [Probabilistically Robust Learning: Balancing Average- and Worst-case Performance](https://mlsanity.com/paper/2202.01136/) (2022)
- [Learning Bounds for Risk-sensitive Learning](https://mlsanity.com/paper/2006.08138/) (2020)
- [Algorithms of Robust Stochastic Optimization Based on Mirror Descent Method](https://mlsanity.com/paper/1907.02707/) (2019)
- [Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimization](https://mlsanity.com/paper/2303.13087/) (2023)
- [Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias](https://mlsanity.com/paper/2210.02720/) (2022)
- [Learning with risks based on M-location](https://mlsanity.com/paper/2012.02424/) (2020)
- [Flexible risk design using bi-directional dispersion](https://mlsanity.com/paper/2203.14434/) (2022)
- [Soft ascent-descent as a stable and flexible alternative to flooding](https://mlsanity.com/paper/2310.10006/) (2023)
- [Classification using margin pursuit](https://mlsanity.com/paper/1810.04863/) (2018)

## Abstract (arXiv)

While the traditional formulation of machine learning tasks is in terms of performance on average, in practice we are often interested in how well a trained model performs on rare or difficult data points at test time. To achieve more robust and balanced generalization, methods applying sharpness-aware minimization to a subset of worst-case examples have proven successful for image classification tasks, but only using overparameterized neural networks under which the relative difference between "easy" and "hard" data points becomes negligible. In this work, we show how such a strategy can dramatically break down under simpler models where the difficulty gap becomes more extreme. As a more flexible alternative, instead of typical sharpness, we propose and evaluate a training criterion which penalizes poor loss concentration, which can be easily combined with loss transformations such exponential tilting, conditional value-at-risk (CVaR), or distributionally robust optimization (DRO) that control tail emphasis.