mlsanityData to 13 Oct 2026

Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook

arXiv 2312.05391 ↗First posted 8 Dec 2023cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

Semantic image segmentation, the process of classifying each pixel in an image into a particular class, plays an important role in many visual understanding systems. As the predominant criterion for evaluating the performance of statistical models, loss functions are crucial for shaping the development of deep learning-based segmentation algorithms and improving their overall performance. To aid researchers in identifying the optimal loss function for their particular application, this survey provides a comprehensive and unified review of 25 loss functions utilized in image segmentation. We provide a novel taxonomy and thorough review of how these loss functions are customized and leveraged in image segmentation, with a systematic categorization emphasizing their significant features and applications. Furthermore, to evaluate the efficacy of these methods in real-world scenarios, we propose unbiased evaluations of some distinct and renowned loss functions on established medical and natural image datasets. We conclude this review by identifying current challenges and unveiling future research opportunities. Finally, we have compiled the reviewed studies that have open-source implementations on our GitHub page.

Key facts. Cited by 33 papers in the map (data through 13 Oct 2026). Builds on 54 of its 71 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

54 of 71 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,435
  5. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,554
  6. 2018Panoptic Segmentationcited by 798
  7. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  8. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 574
  9. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  10. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  11. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 419
  12. 2019Panoptic-DeepLabcited by 362
38 more
  1. 2018nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentationcited by 291
  2. 2022Diffusion Models for Medical Image Analysis: A Comprehensive Surveycited by 273
  3. 2017Tversky loss function for image segmentation using 3D fully convolutional deep networkscited by 228
  4. 2020A survey of loss functions for semantic segmentationcited by 216
  5. 2022Medical Image Segmentation Review: The success of U-Netcited by 191
  6. 2018Boundary loss for highly unbalanced segmentationcited by 187
  7. 2018A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentationcited by 171
  8. 2023MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentationcited by 161
  9. 2014Simultaneous Detection and Segmentationcited by 158
  10. 2019Deep Semantic Segmentation of Natural and Medical Images: A Reviewcited by 125
  11. 2019Reducing the Hausdorff Distance in Medical Image Segmentation with Convolutional Neural Networkscited by 116
  12. 2020EfficientPS: Efficient Panoptic Segmentationcited by 111
  13. 2021MISSFormer: An Effective Medical Image Segmentation Transformercited by 103
  14. 2022HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentationcited by 94
  15. 2019Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentationcited by 88
  16. 2021Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentationcited by 88
  17. 2023Implicit Neural Representation in Medical Imaging: A Comparative Surveycited by 88
  18. 2018Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentationcited by 76
  19. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 76
  20. 2021InverseForm: A Loss Function for Structured Boundary-Aware Segmentationcited by 65
  21. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  22. 2019Region Mutual Information Loss for Semantic Segmentationcited by 49
  23. 20183D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizescited by 42
  24. 2021Active Boundary Loss for Semantic Segmentationcited by 40
  25. 2023Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentationcited by 34
  26. 2020DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentationcited by 33
  27. 2022Contextual Attention Network: Transformer Meets U-Netcited by 33
  28. 2019Distance Map Loss Penalty Term for Semantic Segmentationcited by 27
  29. 2019Global Aggregation then Local Distribution in Fully Convolutional Networkscited by 22
  30. 2021Semi-supervised Multi-task Learning for Semantics and Depthcited by 22
  31. 2016Bridging Category-level and Instance-level Semantic Image Segmentationcited by 21
  32. 2017Loss Max-Pooling for Semantic Image Segmentationcited by 21
  33. 2017Generalised Wasserstein Dice Score for Imbalanced Multi-class Segmentation using Holistic Convolutional Networkscited by 21
  34. 2022PHTrans: Parallelly Aggregating Global and Local Representations for Medical Image Segmentationcited by 16
  35. 2023Robust T-Loss for Medical Image Segmentationcited by 14
  36. 2023Boundary Difference Over Union Loss For Medical Image Segmentationcited by 13
  37. 2021HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentationcited by 12
  38. 2023Topology-Aware Focal Loss for 3D Image Segmentationcited by 12

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 12.

33 papers.

Sort:

By area:

  1. 2024UU-Mamba: Uncertainty-aware U-Mamba for Cardiac Image Segmentationcited by 15
  2. 2025Conformal Prediction for Image Segmentation Using Morphological Prediction Setscited by 13
  3. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  4. 2025FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mappingcited by 6
  5. 2026MambaLiteUNet: Cross-Gated Adaptive Feature Fusion for Robust Skin Lesion Segmentationcited by 4
  6. 2026Sapiens2cited by 4
  7. 2024An Uncertainty-Aware Generalization Framework for Cardiovascular Image Segmentationcited by 3
  8. 2025Dual-Task Learning for Dead Tree Detection and Segmentation with Hybrid Self-Attention U-Nets in Aerial Imagerycited by 3
  9. 2026Mapping Tomato Cropping Systems in California Using AlphaEarth Geospatial Embeddings and Deep Learning Analysiscited by 3
  10. 2025GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRIcited by 2
  11. 2025Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentationcited by 1
  12. 2025Steerable Pyramid Weighted Loss: Multi-Scale Adaptive Weighting for Semantic Segmentationcited by 1
21 more
  1. 2025Task-based Loss Functions in Computer Vision: A Comprehensive Reviewcited by 1
  2. 2025A drone-based framework for coral habitat mapping via weakly supervised segmentationcited by 1
  3. 2026From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tamperingcited by 1
  4. 2026Exploring the potential of AlphaEarth and TESSERA embeddings for Fine-scale Local Climate Zone Mapping: A case study across five cities in Switzerlandcited by 1
  5. 2026Dual-Domain Cross-Modal Decoding for Clinical Text-Guided Medical Image Segmentationcited by 1
  6. 2024Datacube segmentation via Deep Spectral Clusteringcited by 0
  7. 2024Deep Bayesian segmentation for colon polyps: Well-calibrated predictions in medical imagingcited by 0
  8. 2025FloPE: Flower Pose Estimation for Precision Pollinationcited by 0
  9. 2025Transport-Related Surface Detection with Machine Learning: Analyzing Temporal Trends in Madrid and Viennacited by 0
  10. 2025MultiTaskVIF: Segmentation-oriented visible and infrared image fusion via multi-task learningcited by 0
  11. 2025U-NetMN and SegNetMN: Modified U-Net and SegNet models for bimodal SAR image segmentationcited by 0
  12. 2025Microsurgical Instrument Segmentation for Robot-Assisted Surgerycited by 0
  13. 2025Monotonic Transformation Invariant Multi-task Learningcited by 0
  14. 2025Surpassing state of the art on AMD area estimation from RGB fundus images through careful selection of U-Net architectures and loss functions for class imbalancecited by 0
  15. 2025Moment and Highlight Detection via MLLM Frame Segmentationcited by 0
  16. 2026Multi-View Consistent Wound Segmentation With Neural Fieldscited by 0
  17. 2026MonoUNet: A Robust Tiny Neural Network for Automated Knee Cartilage Segmentation on Point-of-Care Ultrasound Devicescited by 0
  18. 2026Component-Adaptive and Lesion-Level Supervision for Improved Small Structure Segmentation in Brain MRIcited by 0
  19. 2026An Uncertainty-Aware Loss Function Incorporating Fuzzy Logic: Application to MRI Brain Image Segmentationcited by 0
  20. 2026Trajectory-Agnostic Asteroid Detection in TESS with Deep Learningcited by 0
  21. 2026GET: Generative Embedding Translation for Medical Image Segmentationcited by 0