# Striking the Right Balance: Recall Loss for Semantic Segmentation arXiv: https://arxiv.org/abs/2106.14917 · HTML: https://mlsanity.com/paper/2106.14917/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 24 references that are in the map. - First posted on arXiv 28 Jun 2021. ## Builds on (most cited first, 18 of 18) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016) - [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017) - [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015) - [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019) - [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019) - [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017) - [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016) - [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014) - [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014) - [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019) - [Tversky loss function for image segmentation using 3D fully convolutional deep networks](https://mlsanity.com/paper/1706.05721/) (2017) - [Metrics for Multi-Class Classification: an Overview](https://mlsanity.com/paper/2008.05756/) (2020) - [Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation](https://mlsanity.com/paper/1805.02798/) (2018) - [Posterior Re-calibration for Imbalanced Datasets](https://mlsanity.com/paper/2010.11820/) (2020) - [Application of Decision Rules for Handling Class Imbalance in Semantic Segmentation](https://mlsanity.com/paper/1901.08394/) (2019) - [Incorporating Depth into both CNN and CRF for Indoor Semantic Segmentation](https://mlsanity.com/paper/1705.07383/) (2017) ## Built on by (most cited first, 19 of 19) - [Balancing Logit Variation for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2306.02061/) (2023) - [Delving into Semantic Scale Imbalance](https://mlsanity.com/paper/2212.14613/) (2022) - [Unsupervised Domain Adaptation for Semantic Segmentation with Pseudo Label Self-Refinement](https://mlsanity.com/paper/2310.16979/) (2023) - [FreDSNet: Joint Monocular Depth and Semantic Segmentation with Fast Fourier Convolutions](https://mlsanity.com/paper/2210.01595/) (2022) - [SSDNeRF: Semantic Soft Decomposition of Neural Radiance Fields](https://mlsanity.com/paper/2212.03406/) (2022) - [Hybrid-Segmentor: A Hybrid Approach to Automated Fine-Grained Crack Segmentation in Civil Infrastructure](https://mlsanity.com/paper/2409.02866/) (2024) - [Enhancing Large Vision Model in Street Scene Semantic Understanding through Leveraging Posterior Optimization Trajectory](https://mlsanity.com/paper/2501.01710/) (2025) - [Unsupervised Domain Adaptation Architecture Search with Self-Training for Land Cover Mapping](https://mlsanity.com/paper/2404.14704/) (2024) - [False Negative Reduction in Semantic Segmentation under Domain Shift using Depth Estimation](https://mlsanity.com/paper/2207.03513/) (2022) - [Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation](https://mlsanity.com/paper/2306.10122/) (2023) - [DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation](https://mlsanity.com/paper/2504.09814/) (2025) - [Dynamic loss balancing and sequential enhancement for road-safety assessment and traffic scene classification](https://mlsanity.com/paper/2211.04165/) (2022) - [JetSeg: Efficient Real-Time Semantic Segmentation Model for Low-Power GPU-Embedded Systems](https://mlsanity.com/paper/2305.11419/) (2023) - [PULASki: Learning inter-rater variability using statistical distances to improve probabilistic segmentation](https://mlsanity.com/paper/2312.15686/) (2023) - [Mixed-domain Training Improves Multi-Mission Terrain Segmentation](https://mlsanity.com/paper/2209.13674/) (2022) - [Reformulating van Rijsbergen's Fβ metric for weighted binary cross-entropy](https://mlsanity.com/paper/2210.16458/) (2022) - [FedDSR: Federated Deep Supervision and Regularization Towards Autonomous Driving](https://mlsanity.com/paper/2512.06676/) (2025) - [MeTok: An Efficient Meteorological Tokenization with Hyper-Aligned Group Learning for Precipitation Nowcasting](https://mlsanity.com/paper/2603.13752/) (2026) - [SenBen: Sensitive Scene Graphs for Explainable Content Moderation](https://mlsanity.com/paper/2604.08819/) (2026) ## Abstract (arXiv) Class imbalance is a fundamental problem in computer vision applications such as semantic segmentation. Specifically, uneven class distributions in a training dataset often result in unsatisfactory performance on under-represented classes. Many works have proposed to weight the standard cross entropy loss function with pre-computed weights based on class statistics, such as the number of samples and class margins. There are two major drawbacks to these methods: 1) constantly up-weighting minority classes can introduce excessive false positives in semantic segmentation; 2) a minority class is not necessarily a hard class. The consequence is low precision due to excessive false positives. In this regard, we propose a hard-class mining loss by reshaping the vanilla cross entropy loss such that it weights the loss for each class dynamically based on instantaneous recall performance. We show that the novel recall loss changes gradually between the standard cross entropy loss and the inverse frequency weighted loss. Recall loss also leads to improved mean accuracy while offering competitive mean Intersection over Union (IoU) performance. On Synthia dataset, recall loss achieves 9% relative improvement on mean accuracy with competitive mean IoU using DeepLab-ResNet18 compared to the cross entropy loss. Code available at https://github.com/PotatoTian/recall-semseg.