# Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook

arXiv: https://arxiv.org/abs/2312.05391 · HTML: https://mlsanity.com/paper/2312.05391/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 71 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Tversky loss function for image segmentation using 3D fully convolutional deep networks](https://mlsanity.com/paper/1706.05721/) (2017)
- [A survey of loss functions for semantic segmentation](https://mlsanity.com/paper/2006.14822/) (2020)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [Boundary loss for highly unbalanced segmentation](https://mlsanity.com/paper/1812.07032/) (2018)
- [A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentation](https://mlsanity.com/paper/1810.07842/) (2018)
- [MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation](https://mlsanity.com/paper/2303.09975/) (2023)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [Reducing the Hausdorff Distance in Medical Image Segmentation with Convolutional Neural Networks](https://mlsanity.com/paper/1904.10030/) (2019)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [MISSFormer: An Effective Medical Image Segmentation Transformer](https://mlsanity.com/paper/2109.07162/) (2021)
- [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2207.08518/) (2022)
- [Implicit Neural Representation in Medical Imaging: A Comparative Survey](https://mlsanity.com/paper/2307.16142/) (2023)
- [Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation](https://mlsanity.com/paper/1904.06346/) (2019)
- [Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation](https://mlsanity.com/paper/2102.04525/) (2021)
- [Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation](https://mlsanity.com/paper/1805.02798/) (2018)

## Built on by (most cited first, 30 of 34)

- [UU-Mamba: Uncertainty-aware U-Mamba for Cardiac Image Segmentation](https://mlsanity.com/paper/2405.17496/) (2024)
- [Conformal Prediction for Image Segmentation Using Morphological Prediction Sets](https://mlsanity.com/paper/2503.05618/) (2025)
- [Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scene](https://mlsanity.com/paper/2503.15019/) (2025)
- [FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping](https://mlsanity.com/paper/2506.07080/) (2025)
- [MambaLiteUNet: Cross-Gated Adaptive Feature Fusion for Robust Skin Lesion Segmentation](https://mlsanity.com/paper/2604.20286/) (2026)
- [Sapiens2](https://mlsanity.com/paper/2604.21681/) (2026)
- [An Uncertainty-Aware Generalization Framework for Cardiovascular Image Segmentation](https://mlsanity.com/paper/2409.14305/) (2024)
- [Dual-Task Learning for Dead Tree Detection and Segmentation with Hybrid Self-Attention U-Nets in Aerial Imagery](https://mlsanity.com/paper/2503.21438/) (2025)
- [Mapping Tomato Cropping Systems in California Using AlphaEarth Geospatial Embeddings and Deep Learning Analysis](https://mlsanity.com/paper/2605.21804/) (2026)
- [GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI](https://mlsanity.com/paper/2503.04325/) (2025)
- [Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation](https://mlsanity.com/paper/2502.00563/) (2025)
- [Steerable Pyramid Weighted Loss: Multi-Scale Adaptive Weighting for Semantic Segmentation](https://mlsanity.com/paper/2503.06604/) (2025)
- [Task-based Loss Functions in Computer Vision: A Comprehensive Review](https://mlsanity.com/paper/2504.04242/) (2025)
- [A drone-based framework for coral habitat mapping via weakly supervised segmentation](https://mlsanity.com/paper/2508.18958/) (2025)
- [From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering](https://mlsanity.com/paper/2603.20193/) (2026)
- [Exploring the potential of AlphaEarth and TESSERA embeddings for Fine-scale Local Climate Zone Mapping: A case study across five cities in Switzerland](https://mlsanity.com/paper/2606.20034/) (2026)
- [Dual-Domain Cross-Modal Decoding for Clinical Text-Guided Medical Image Segmentation](https://mlsanity.com/paper/2608.11335/) (2026)
- [Datacube segmentation via Deep Spectral Clustering](https://mlsanity.com/paper/2401.17695/) (2024)
- [Deep Bayesian segmentation for colon polyps: Well-calibrated predictions in medical imaging](https://mlsanity.com/paper/2407.16608/) (2024)
- [FloPE: Flower Pose Estimation for Precision Pollination](https://mlsanity.com/paper/2503.11692/) (2025)
- [Transport-Related Surface Detection with Machine Learning: Analyzing Temporal Trends in Madrid and Vienna](https://mlsanity.com/paper/2503.15653/) (2025)
- [MultiTaskVIF: Segmentation-oriented visible and infrared image fusion via multi-task learning](https://mlsanity.com/paper/2505.06665/) (2025)
- [U-NetMN and SegNetMN: Modified U-Net and SegNet models for bimodal SAR image segmentation](https://mlsanity.com/paper/2506.05444/) (2025)
- [Microsurgical Instrument Segmentation for Robot-Assisted Surgery](https://mlsanity.com/paper/2509.11727/) (2025)
- [Monotonic Transformation Invariant Multi-task Learning](https://mlsanity.com/paper/2509.23948/) (2025)
- [Surpassing state of the art on AMD area estimation from RGB fundus images through careful selection of U-Net architectures and loss functions for class imbalance](https://mlsanity.com/paper/2510.26778/) (2025)
- [Moment and Highlight Detection via MLLM Frame Segmentation](https://mlsanity.com/paper/2512.12246/) (2025)
- [Multi-View Consistent Wound Segmentation With Neural Fields](https://mlsanity.com/paper/2601.16487/) (2026)
- [MonoUNet: A Robust Tiny Neural Network for Automated Knee Cartilage Segmentation on Point-of-Care Ultrasound Devices](https://mlsanity.com/paper/2604.07780/) (2026)
- [Component-Adaptive and Lesion-Level Supervision for Improved Small Structure Segmentation in Brain MRI](https://mlsanity.com/paper/2604.08015/) (2026)

## Abstract (arXiv)

Semantic image segmentation, the process of classifying each pixel in an image into a particular class, plays an important role in many visual understanding systems. As the predominant criterion for evaluating the performance of statistical models, loss functions are crucial for shaping the development of deep learning-based segmentation algorithms and improving their overall performance. To aid researchers in identifying the optimal loss function for their particular application, this survey provides a comprehensive and unified review of 25 loss functions utilized in image segmentation. We provide a novel taxonomy and thorough review of how these loss functions are customized and leveraged in image segmentation, with a systematic categorization emphasizing their significant features and applications. Furthermore, to evaluate the efficacy of these methods in real-world scenarios, we propose unbiased evaluations of some distinct and renowned loss functions on established medical and natural image datasets. We conclude this review by identifying current challenges and unveiling future research opportunities. Finally, we have compiled the reviewed studies that have open-source implementations on our GitHub page.