# Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning

arXiv: https://arxiv.org/abs/2011.05231 · HTML: https://mlsanity.com/paper/2011.05231/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 10 of 10)

- [The Effect of Model Compression on Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2201.01709/) (2022)
- [FundusQ-Net: a Regression Quality Assessment Deep Learning Algorithm for Fundus Images Quality Grading](https://mlsanity.com/paper/2205.01676/) (2022)
- [SubStrat: A Subset-Based Strategy for Faster AutoML](https://mlsanity.com/paper/2206.03070/) (2022)
- [Towards noise contrastive estimation with soft targets for conditional models](https://mlsanity.com/paper/2404.14076/) (2024)
- [Data Augmentation for Compositional Data: Advancing Predictive Models of the Microbiome](https://mlsanity.com/paper/2205.09906/) (2022)
- [On the Normalizing Constant of the Continuous Categorical Distribution](https://mlsanity.com/paper/2204.13290/) (2022)
- [DSC-IITISM at FinCausal 2021: Combining POS tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents](https://mlsanity.com/paper/2111.00490/) (2021)
- [An Artificial Neural Network Functionalized by Evolution](https://mlsanity.com/paper/2205.10118/) (2022)
- [Interpretable Computer Vision Models through Adversarial Training: Unveiling the Robustness-Interpretability Connection](https://mlsanity.com/paper/2307.02500/) (2023)
- [DAIRE: A lightweight AI model for real-time detection of Controller Area Network attacks in the Internet of Vehicles](https://mlsanity.com/paper/2604.20771/) (2026)

## Abstract (arXiv)

Modern deep learning is primarily an experimental science, in which empirical advances occasionally come at the expense of probabilistic rigor. Here we focus on one such example; namely the use of the categorical cross-entropy loss to model data that is not strictly categorical, but rather takes values on the simplex. This practice is standard in neural network architectures with label smoothing and actor-mimic reinforcement learning, amongst others. Drawing on the recently discovered continuous-categorical distribution, we propose probabilistically-inspired alternatives to these models, providing an approach that is more principled and theoretically appealing. Through careful experimentation, including an ablation study, we identify the potential for outperformance in these models, thereby highlighting the importance of a proper probabilistic treatment, as well as illustrating some of the failure modes thereof.