# EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification

arXiv: https://arxiv.org/abs/2409.00908 · HTML: https://mlsanity.com/paper/2409.00908/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 72 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Handbook of Convergence Theorems for (Stochastic) Gradient Methods](https://mlsanity.com/paper/2301.11235/) (2023)
- [Understanding data augmentation for classification: when to warp?](https://mlsanity.com/paper/1609.08764/) (2016)
- [A Unified Theory of Diversity in Ensemble Learning](https://mlsanity.com/paper/2301.03962/) (2023)
- [Convergence rates for the stochastic gradient descent method for non-convex objective functions](https://mlsanity.com/paper/1904.01517/) (2019)
- [Meta-Learning via Learned Loss](https://mlsanity.com/paper/1906.05374/) (2019)
- [The Tree Ensemble Layer: Differentiability meets Conditional Computation](https://mlsanity.com/paper/2002.07772/) (2020)
- [Random-projection ensemble classification](https://mlsanity.com/paper/1504.04595/) (2015)
- [Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation](https://mlsanity.com/paper/1601.03651/) (2016)
- [RankSEG: A Consistent Ranking-based Framework for Segmentation](https://mlsanity.com/paper/2206.13086/) (2022)
- [On Classification-Calibration of Gamma-Phi Losses](https://mlsanity.com/paper/2302.07321/) (2023)
- [Meta-Learning Loss Functions for Deep Neural Networks](https://mlsanity.com/paper/2406.09713/) (2024)

## Built on by (most cited first, 1 of 1)

- [Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget](https://mlsanity.com/paper/2606.16110/) (2026)

## Abstract (arXiv)

Empirical risk minimization (ERM) with a computationally feasible surrogate loss is a widely accepted approach for classification. Notably, the convexity and calibration (CC) properties of a loss function ensure consistency of ERM in maximizing accuracy, thereby offering a wide range of options for surrogate losses. In this article, we propose a novel ensemble method, namely EnsLoss, which extends the ensemble learning concept to combine loss functions within the ERM framework. A key feature of our method is the consideration on preserving the "legitimacy" of the combined losses, i.e., ensuring the CC properties. Specifically, we first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions and directly generating calibrated loss-derivatives. Therefore, inspired by Dropout, EnsLoss enables loss ensembles through one training process with doubly stochastic gradient descent (i.e., random batch samples and random calibrated loss-derivatives). We theoretically establish the statistical consistency of our approach and provide insights into its benefits. The numerical effectiveness of EnsLoss compared to fixed loss methods is demonstrated through experiments on a broad range of 14 OpenML tabular datasets and 46 image datasets with various deep learning architectures. Python repository and source code are available on GitHub at https://github.com/statmlben/ensloss.