# On Loss Functions for Deep Neural Networks in Classification

arXiv: https://arxiv.org/abs/1702.05659 · HTML: https://mlsanity.com/paper/1702.05659/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2017.

## Built on by (most cited first, 30 of 68)

- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Theoretical Guarantees for Permutation-Equivariant Quantum Neural Networks](https://mlsanity.com/paper/2210.09974/) (2022)
- [Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs](https://mlsanity.com/paper/2212.09034/) (2022)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionality](https://mlsanity.com/paper/2105.08675/) (2021)
- [An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift](https://mlsanity.com/paper/2202.12123/) (2022)
- [Camouflaged Instance Segmentation In-The-Wild: Dataset, Method, and Benchmark Suite](https://mlsanity.com/paper/2103.17123/) (2021)
- [Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training](https://mlsanity.com/paper/2103.00673/) (2021)
- [A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate](https://mlsanity.com/paper/2102.02617/) (2021)
- [Universal Spam Detection using Transfer Learning of BERT Model](https://mlsanity.com/paper/2202.03480/) (2022)
- [MambaEVT: Event Stream based Visual Object Tracking using State Space Model](https://mlsanity.com/paper/2408.10487/) (2024)
- [Score-oriented loss (SOL) functions](https://mlsanity.com/paper/2103.15522/) (2021)
- [Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity](https://mlsanity.com/paper/2112.14869/) (2021)
- [Face Identity-Aware Disentanglement in StyleGAN](https://mlsanity.com/paper/2309.12033/) (2023)
- [LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning](https://mlsanity.com/paper/2410.00645/) (2024)
- [Tilting the playing field: Dynamical loss functions for machine learning](https://mlsanity.com/paper/2102.03793/) (2021)
- [Meta Ordinal Regression Forest for Medical Image Classification with Ordinal Labels](https://mlsanity.com/paper/2203.07725/) (2022)
- [Surface EMG-Based Inter-Session/Inter-Subject Gesture Recognition by Leveraging Lightweight All-ConvNet and Transfer Learning](https://mlsanity.com/paper/2305.08014/) (2023)
- [General Loss Functions Lead to (Approximate) Interpolation in High Dimensions](https://mlsanity.com/paper/2303.07475/) (2023)
- [Convolutional Neural Nets in Chemical Engineering: Foundations, Computations, and Applications](https://mlsanity.com/paper/2101.04869/) (2021)
- [Neural ODE to model and prognose thermoacoustic instability](https://mlsanity.com/paper/2106.12758/) (2021)
- [Domain Adaptation in Multilingual and Multi-Domain Monolingual Settings for Complex Word Identification](https://mlsanity.com/paper/2205.07283/) (2022)
- [A comprehensive theoretical framework for the optimization of neural networks classification performance with respect to weighted metrics](https://mlsanity.com/paper/2305.13472/) (2023)
- [AnyLoss: Transforming Classification Metrics into Loss Functions](https://mlsanity.com/paper/2405.14745/) (2024)
- [Towards Efficient and Exact Forgetting Services in Pre-Trained-Model-based Continual Learning](https://mlsanity.com/paper/2505.12239/) (2025)
- [AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Data](https://mlsanity.com/paper/2505.12245/) (2025)
- [Voice Cloning: a Multi-Speaker Text-to-Speech Synthesis Approach based on Transfer Learning](https://mlsanity.com/paper/2102.05630/) (2021)
- [Food Recipe Recommendation Based on Ingredients Detection Using Deep Learning](https://mlsanity.com/paper/2203.06721/) (2022)
- [Analysis of the performance of U-Net neural networks for the segmentation of living cells](https://mlsanity.com/paper/2210.01538/) (2022)

## Abstract (arXiv)

Deep neural networks are currently among the most commonly used classifiers. Despite easily achieving very good performance, one of the best selling points of these models is their modular design - one can conveniently adapt their architecture to specific needs, change connectivity patterns, attach specialised layers, experiment with a large amount of activation functions, normalisation schemes and many others. While one can find impressively wide spread of various configurations of almost every aspect of the deep nets, one element is, in authors' opinion, underrepresented - while solving classification problems, vast majority of papers and applications simply use log loss. In this paper we try to investigate how particular choices of loss functions affect deep models and their learning dynamics, as well as resulting classifiers robustness to various effects. We perform experiments on classical datasets, as well as provide some additional, theoretical insights into the problem. In particular we show that L1 and L2 losses are, quite surprisingly, justified classification objectives for deep nets, by providing probabilistic interpretation in terms of expected misclassification. We also introduce two losses which are not typically used as deep nets objectives and show that they are viable alternatives to the existing ones.