# Cut your Losses with Squentropy

arXiv: https://arxiv.org/abs/2302.03952 · HTML: https://mlsanity.com/paper/2302.03952/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 8 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Large Margin Deep Networks for Classification](https://mlsanity.com/paper/1803.05598/) (2018)
- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks](https://mlsanity.com/paper/1910.01663/) (2019)
- [Optimization for deep learning: theory and algorithms](https://mlsanity.com/paper/1912.08957/) (2019)

## Built on by (most cited first, 6 of 6)

- [Pearls from Pebbles: Improved Confidence Functions for Auto-labeling](https://mlsanity.com/paper/2404.16188/) (2024)
- [Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers](https://mlsanity.com/paper/2601.02543/) (2026)
- [A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initialization](https://mlsanity.com/paper/2503.06385/) (2025)
- [Accelerated Neural Network Training with Rooted Logistic Objectives](https://mlsanity.com/paper/2310.03890/) (2023)
- [Minimizing Chebyshev Prototype Risk Magically Mitigates the Perils of Overfitting](https://mlsanity.com/paper/2404.07083/) (2024)
- [One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Context](https://mlsanity.com/paper/2609.01311/) (2026)

## Abstract (arXiv)

Nearly all practical neural models for classification are trained using cross-entropy loss. Yet this ubiquitous choice is supported by little theoretical or empirical evidence. Recent work (Hui & Belkin, 2020) suggests that training using the (rescaled) square loss is often superior in terms of the classification accuracy. In this paper we propose the "squentropy" loss, which is the sum of two terms: the cross-entropy loss and the average square loss over the incorrect classes. We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy. We also demonstrate that it provides significantly better model calibration than either of these alternative losses and, furthermore, has less variance with respect to the random initialization. Additionally, in contrast to the square loss, squentropy loss can typically be trained using exactly the same optimization parameters, including the learning rate, as the standard cross-entropy loss, making it a true "plug-and-play" replacement. Finally, unlike the rescaled square loss, multiclass squentropy contains no parameters that need to be adjusted.