mlsanityData to 14 Oct 2026

A survey and taxonomy of loss functions in machine learning

arXiv 2301.05579 ↗First posted 13 Jan 2023cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Most state-of-the-art machine learning techniques revolve around the optimisation of loss functions. Defining appropriate loss functions is therefore critical to successfully solving problems in this field. In this survey, we present a comprehensive overview of the most widely used loss functions across key applications, including regression, classification, generative modeling, ranking, and energy-based modeling. We introduce 43 distinct loss functions, structured within an intuitive taxonomy that clarifies their theoretical foundations, properties, and optimal application contexts. This survey is intended as a resource for undergraduate, graduate, and Ph.D. students, as well as researchers seeking a deeper understanding of loss functions.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 43 of its 145 references that are in the map. First posted on arXiv 13 Jan 2023.

Builds on

43 of 145 references.

References · 43

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2016Density estimation using Real NVPcited by 2,042
  10. 2016node2vec: Scalable Feature Learning for Networkscited by 1,918
  11. 2014DeepWalk: Online Learning of Social Representationscited by 1,547
  12. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
31 more
  1. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  2. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  3. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  4. 2016Pixel Recurrent Neural Networkscited by 816
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  6. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  7. 2014Deep metric learning using Triplet networkcited by 422
  8. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  10. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  11. 2021How to Train Your Energy-Based Modelscited by 274
  12. 2020A survey of loss functions for semantic segmentationcited by 217
  13. 2017struc2vec: Learning Node Representations from Structural Identitycited by 215
  14. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 190
  15. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  16. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  17. 2016Variational Autoencoder for Deep Learning of Images, Labels and Captionscited by 137
  18. 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 114
  19. 2016Higher-order organization of complex networkscited by 110
  20. 2020Training robust neural networks using Lipschitz boundscited by 99
  21. 2017Learning Structural Node Embeddings Via Diffusion Waveletscited by 97
  22. 2018Adversarially Regularized Graph Autoencoder for Graph Embeddingcited by 84
  23. 2019Maximum Entropy Generators for Energy-Based Modelscited by 56
  24. 2018Attention Models in Graphs: A Surveycited by 41
  25. 2019Model Agnostic Contrastive Explanations for Structured Datacited by 41
  26. 2017Improving Pairwise Ranking for Multi-label Image Classificationcited by 40
  27. 2017A survey of robot learning from demonstrations for Human-Robot Collaborationcited by 38
  28. 2021Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)cited by 35
  29. 2019Model Based Planning with Energy Based Modelscited by 33
  30. 2019A Survey on Deep Learning-based Non-Invasive Brain Signals:Recent Advances and New Frontierscited by 10
  31. 2013Subgraph Frequencies: Mapping the Empirical and Extremal Geography of Large Graph Collectionscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 2.

15 papers.

Sort:

  1. 2024A Pattern Language for Machine Learning Taskscited by 4
  2. 2024Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformercited by 4
  3. 20253D Neural Operator-Based Flow Surrogates around 3D geometries: Signed Distance Functions and Derivative Constraintscited by 3
  4. 2025Neuro-Argumentative Learning with Case-Based Reasoningcited by 3
  5. 2025Beyond Regularity: Modeling Chaotic Mobility Patterns for Next Location Predictioncited by 3
  6. 2024Gradient Aligned Regression via Pairwise Lossescited by 2
  7. 2024Meta-learning Loss Functions of Parametric Partial Differential Equations Using Physics-Informed Neural Networkscited by 2
  8. 2024IID Relaxation by Logical Expressivity: A Research Agenda for Fitting Logics to Neurosymbolic Requirementscited by 1
  9. 2025Fine Tuning a Simulation-Driven Estimatorcited by 1
  10. 2023AI Powered Road Network Prediction with Multi-Modal Datacited by 0
  11. 2025Including local feature interactions in deep non-negative matrix factorization networks improves performancecited by 0
  12. 2025XL-DURel: Finetuning Sentence Transformers for Ordinal Word-in-Context Classificationcited by 0
3 more
  1. 2025Bulk-boundary decomposition of neural networkscited by 0
  2. 2026Ratio-based Loss Functionscited by 0
  3. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0