# On the Separability of Classes with the Cross-Entropy Loss Function

arXiv: https://arxiv.org/abs/1909.06930 · HTML: https://mlsanity.com/paper/1909.06930/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 8 of 8)

- [FLAC: Fairness-Aware Representation Learning by Suppressing Attribute-Class Associations](https://mlsanity.com/paper/2304.14252/) (2023)
- [Contrastive Unlearning: A Contrastive Approach to Machine Unlearning](https://mlsanity.com/paper/2401.10458/) (2024)
- [Debiasing Machine Unlearning with Counterfactual Examples](https://mlsanity.com/paper/2404.15760/) (2024)
- [NCDD: Nearest Centroid Distance Deficit for Out-Of-Distribution Detection in Gastrointestinal Vision](https://mlsanity.com/paper/2412.01590/) (2024)
- [Node-level Contrastive Unlearning on Graph Neural Networks](https://mlsanity.com/paper/2503.02959/) (2025)
- [Meta Pattern Concern Score: A Novel Evaluation Measure with Human Values for Multi-classifiers](https://mlsanity.com/paper/2209.06408/) (2022)
- [Preserving Cross-Modal Stability for Visual Unlearning in Multimodal Scenarios](https://mlsanity.com/paper/2509.23895/) (2025)
- [CephRes-MHNet: A Multi-Head Residual Network for Accurate and Robust Cephalometric Landmark Detection](https://mlsanity.com/paper/2511.10173/) (2025)

## Abstract (arXiv)

In this paper, we focus on the separability of classes with the cross-entropy loss function for classification problems by theoretically analyzing the intra-class distance and inter-class distance (i.e. the distance between any two points belonging to the same class and different classes, respectively) in the feature space, i.e. the space of representations learnt by neural networks. Specifically, we consider an arbitrary network architecture having a fully connected final layer with Softmax activation and trained using the cross-entropy loss. We derive expressions for the value and the distribution of the squared L2 norm of the product of a network dependent matrix and a random intra-class and inter-class distance vector (i.e. the vector between any two points belonging to the same class and different classes), respectively, in the learnt feature space (or the transformation of the original data) just before Softmax activation, as a function of the cross-entropy loss value. The main result of our analysis is the derivation of a lower bound for the probability with which the inter-class distance is more than the intra-class distance in this feature space, as a function of the loss value. We do so by leveraging some empirical statistical observations with mild assumptions and sound theoretical analysis. As per intuition, the probability with which the inter-class distance is more than the intra-class distance decreases as the loss value increases, i.e. the classes are better separated when the loss value is low. To the best of our knowledge, this is the first work of theoretical nature trying to explain the separability of classes in the feature space learnt by neural networks trained with the cross-entropy loss function.