# Cross Entropy versus Label Smoothing: A Neural Collapse Perspective arXiv: https://arxiv.org/abs/2402.03979 ยท HTML: https://mlsanity.com/paper/2402.03979/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 39 references that are in the map. - First posted on arXiv 6 Feb 2024. ## Builds on (most cited first, 28 of 28) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019) - [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020) - [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017) - [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021) - [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020) - [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021) - [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021) - [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020) - [Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2302.03004/) (2023) - [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022) - [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021) - [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022) - [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022) - [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022) - [Provable approximation properties for deep neural networks](https://mlsanity.com/paper/1509.07385/) (2015) - [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020) - [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022) - [Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning](https://mlsanity.com/paper/2204.08735/) (2022) - [Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data](https://mlsanity.com/paper/2301.00437/) (2023) - [Generalized Neural Collapse for a Large Number of Classes](https://mlsanity.com/paper/2310.05351/) (2023) - [Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data](https://mlsanity.com/paper/2309.09725/) (2023) - [Explicit regularization and implicit bias in deep network classifiers trained with the square loss](https://mlsanity.com/paper/2101.00072/) (2020) - [Perturbation Analysis of Neural Collapse](https://mlsanity.com/paper/2210.16658/) (2022) - [Inducing Neural Collapse in Deep Long-tailed Learning](https://mlsanity.com/paper/2302.12453/) (2023) - [Continual Learning by Modeling Intra-Class Variation](https://mlsanity.com/paper/2210.05398/) (2022) ## Built on by (most cited first, 11 of 11) - [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024) - [The Prevalence of Neural Collapse in Neural Multivariate Regression](https://mlsanity.com/paper/2409.04180/) (2024) - [MaxSup: Overcoming Representation Collapse in Label Smoothing](https://mlsanity.com/paper/2502.15798/) (2025) - [Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model](https://mlsanity.com/paper/2505.09308/) (2025) - [Trojan Cleansing with Neural Collapse](https://mlsanity.com/paper/2411.12914/) (2024) - [Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension](https://mlsanity.com/paper/2510.01105/) (2025) - [From Coefficients to Directions: Rethinking Model Merging with Directional Alignment](https://mlsanity.com/paper/2512.00391/) (2025) - [COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control](https://mlsanity.com/paper/2601.06122/) (2026) - [The Implicit Bias of Logit Regularization](https://mlsanity.com/paper/2602.12039/) (2026) - [The role of class encoding in neural collapse](https://mlsanity.com/paper/2606.00344/) (2026) - [MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization](https://mlsanity.com/paper/2609.30855/) (2026) ## Abstract (arXiv) Label smoothing loss is a widely adopted technique to mitigate overfitting in deep neural networks. This paper studies label smoothing from the perspective of Neural Collapse (NC), a powerful empirical and theoretical framework which characterizes model behavior during the terminal phase of training. We first show empirically that models trained with label smoothing converge faster to neural collapse solutions and attain a stronger level of neural collapse. Additionally, we show that at the same level of NC1, models under label smoothing loss exhibit intensified NC2. These findings provide valuable insights into the performance benefits and enhanced model calibration under label smoothing loss. We then leverage the unconstrained feature model to derive closed-form solutions for the global minimizers for both loss functions and further demonstrate that models under label smoothing have a lower conditioning number and, therefore, theoretically converge faster. Our study, combining empirical evidence and theoretical results, not only provides nuanced insights into the differences between label smoothing and cross-entropy losses, but also serves as an example of how the powerful neural collapse framework can be used to improve our understanding of DNNs.