# Cross Entropy in Deep Learning of Classifiers Is Unnecessary -- ISBE Error is All You Need

arXiv: https://arxiv.org/abs/2311.16357 · HTML: https://mlsanity.com/paper/2311.16357/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Real-Time Flying Object Detection with YOLOv8](https://mlsanity.com/paper/2305.09972/) (2023)
- [MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition](https://mlsanity.com/paper/2004.08531/) (2020)
- [Annotated History of Modern AI and Deep Learning](https://mlsanity.com/paper/2212.11279/) (2022)
- [Multi-modal Residual Perceptron Network for Audio-Video Emotion Recognition](https://mlsanity.com/paper/2107.10742/) (2021)
- [DiaCorrect: Error Correction Back-end For Speaker Diarization](https://mlsanity.com/paper/2309.08377/) (2023)
- [Symbolic Tensor Neural Networks for Digital Media - from Tensor Processing via BNF Graph Rules to CREAMS Applications](https://mlsanity.com/paper/1809.06582/) (2018)

## Abstract (arXiv)

In deep learning classifiers, the cost function usually takes the form of a combination of SoftMax and CrossEntropy functions. The SoftMax unit transforms the scores predicted by the model network into assessments of the degree (probabilities) of an object's membership to a given class. On the other hand, CrossEntropy measures the divergence of this prediction from the distribution of target scores. This work introduces the ISBE functionality, justifying the thesis about the redundancy of cross entropy computation in deep learning of classifiers. Not only can we omit the calculation of entropy, but also, during back-propagation, there is no need to direct the error to the normalization unit for its backward transformation. Instead, the error is sent directly to the model's network. Using examples of perceptron and convolutional networks as classifiers of images from the MNIST collection, it is observed for ISBE that results are not degraded with SoftMax only, but also with other activation functions such as Sigmoid, Tanh, or their hard variants HardSigmoid and HardTanh. Moreover, up to three percent of time is saved within the total time of forward and backward stages. The article is addressed mainly to programmers and students interested in deep model learning. For example, it illustrates in code snippets possible ways to implement ISBE units, but also formally proves that the softmax trick only applies to the class of softmax functions with relocations.