# Evaluating the Impact of Loss Function Variation in Deep Learning for Classification

arXiv: https://arxiv.org/abs/2210.16003 · HTML: https://mlsanity.com/paper/2210.16003/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [Classification vs regression in overparameterized regimes: Does the loss function matter?](https://mlsanity.com/paper/2005.08054/) (2020)
- [On Loss Functions for Deep Neural Networks in Classification](https://mlsanity.com/paper/1702.05659/) (2017)
- [Studying Very Low Resolution Recognition Using Deep Networks](https://mlsanity.com/paper/1601.04153/) (2016)
- [IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters](https://mlsanity.com/paper/1903.12141/) (2019)
- [On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.07281/) (2020)
- [An Ensemble of Simple Convolutional Neural Network Models for MNIST Digit Recognition](https://mlsanity.com/paper/2008.10400/) (2020)
- [Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions](https://mlsanity.com/paper/1901.02302/) (2019)
- [A Comparative Study of Deep Learning Loss Functions for Multi-Label Remote Sensing Image Classification](https://mlsanity.com/paper/2009.13935/) (2020)
- [Heartbeat Classification in Wearables Using Multi-layer Perceptron and Time-Frequency Joint Distribution of ECG](https://mlsanity.com/paper/1908.06865/) (2019)

## Built on by (most cited first, 1 of 1)

- [GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost](https://mlsanity.com/paper/2405.14736/) (2024)

## Abstract (arXiv)

The loss function is arguably among the most important hyperparameters for a neural network. Many loss functions have been designed to date, making a correct choice nontrivial. However, elaborate justifications regarding the choice of the loss function are not made in related work. This is, as we see it, an indication of a dogmatic mindset in the deep learning community which lacks empirical foundation. In this work, we consider deep neural networks in a supervised classification setting and analyze the impact the choice of loss function has onto the training result. While certain loss functions perform suboptimally, our work empirically shows that under-represented losses such as the KL Divergence can outperform the State-of-the-Art choices significantly, highlighting the need to include the loss function as a tuned hyperparameter rather than a fixed choice.