# A survey and taxonomy of loss functions in machine learning

arXiv: https://arxiv.org/abs/2301.05579 · HTML: https://mlsanity.com/paper/2301.05579/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 145 references that are in the map.
- First posted on arXiv 13 Jan 2023.

## Builds on (most cited first, 30 of 43)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [A survey of loss functions for semantic segmentation](https://mlsanity.com/paper/2006.14822/) (2020)
- [struc2vec: Learning Node Representations from Structural Identity](https://mlsanity.com/paper/1704.03165/) (2017)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020)
- [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)

## Built on by (most cited first, 15 of 15)

- [A Pattern Language for Machine Learning Tasks](https://mlsanity.com/paper/2407.02424/) (2024)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [3D Neural Operator-Based Flow Surrogates around 3D geometries: Signed Distance Functions and Derivative Constraints](https://mlsanity.com/paper/2503.17289/) (2025)
- [Neuro-Argumentative Learning with Case-Based Reasoning](https://mlsanity.com/paper/2505.15742/) (2025)
- [Beyond Regularity: Modeling Chaotic Mobility Patterns for Next Location Prediction](https://mlsanity.com/paper/2509.11713/) (2025)
- [Gradient Aligned Regression via Pairwise Losses](https://mlsanity.com/paper/2402.06104/) (2024)
- [Meta-learning Loss Functions of Parametric Partial Differential Equations Using Physics-Informed Neural Networks](https://mlsanity.com/paper/2412.00225/) (2024)
- [IID Relaxation by Logical Expressivity: A Research Agenda for Fitting Logics to Neurosymbolic Requirements](https://mlsanity.com/paper/2404.19485/) (2024)
- [Fine Tuning a Simulation-Driven Estimator](https://mlsanity.com/paper/2504.04480/) (2025)
- [AI Powered Road Network Prediction with Multi-Modal Data](https://mlsanity.com/paper/2312.17040/) (2023)
- [Including local feature interactions in deep non-negative matrix factorization networks improves performance](https://mlsanity.com/paper/2503.20398/) (2025)
- [XL-DURel: Finetuning Sentence Transformers for Ordinal Word-in-Context Classification](https://mlsanity.com/paper/2507.14578/) (2025)
- [Bulk-boundary decomposition of neural networks](https://mlsanity.com/paper/2511.02003/) (2025)
- [Ratio-based Loss Functions](https://mlsanity.com/paper/2605.05808/) (2026)
- [A Unified Framework for In-Context Learning with Causal and Masked Language Models](https://mlsanity.com/paper/2607.04081/) (2026)

## Abstract (arXiv)

Most state-of-the-art machine learning techniques revolve around the optimisation of loss functions. Defining appropriate loss functions is therefore critical to successfully solving problems in this field. In this survey, we present a comprehensive overview of the most widely used loss functions across key applications, including regression, classification, generative modeling, ranking, and energy-based modeling. We introduce 43 distinct loss functions, structured within an intuitive taxonomy that clarifies their theoretical foundations, properties, and optimal application contexts. This survey is intended as a resource for undergraduate, graduate, and Ph.D. students, as well as researchers seeking a deeper understanding of loss functions.