# Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation

arXiv: https://arxiv.org/abs/2010.07930 · HTML: https://mlsanity.com/paper/2010.07930/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)
- [AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks](https://mlsanity.com/paper/2103.14026/) (2021)
- [Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression](https://mlsanity.com/paper/2110.13675/) (2021)
- [Pseudo-Labeled Auto-Curriculum Learning for Semi-Supervised Keypoint Localization](https://mlsanity.com/paper/2201.08613/) (2022)
- [Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels](https://mlsanity.com/paper/2302.05666/) (2023)
- [AutoLossGen: Automatic Loss Function Generation for Recommender Systems](https://mlsanity.com/paper/2204.13160/) (2022)
- [Hybrid-Supervised Dual-Search: Leveraging Automatic Learning for Loss-free Multi-Exposure Image Fusion](https://mlsanity.com/paper/2309.01113/) (2023)
- [LPZero: Language Model Zero-cost Proxy Search from Zero](https://mlsanity.com/paper/2410.04808/) (2024)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search](https://mlsanity.com/paper/2111.05063/) (2021)
- [Problem Learning: Towards the Free Will of Machines](https://mlsanity.com/paper/2109.00177/) (2021)
- [HLIC: Harmonizing Optimization Metrics in Learned Image Compression by Reinforcement Learning](https://mlsanity.com/paper/2109.14863/) (2021)
- [Searching Parameterized AP Loss for Object Detection](https://mlsanity.com/paper/2112.05138/) (2021)
- [BoundMatch: Boundary detection applied to semi-supervised segmentation](https://mlsanity.com/paper/2503.23519/) (2025)

## Abstract (arXiv)

Designing proper loss functions is essential in training deep networks. Especially in the field of semantic segmentation, various evaluation metrics have been proposed for diverse scenarios. Despite the success of the widely adopted cross-entropy loss and its variants, the mis-alignment between the loss functions and evaluation metrics degrades the network performance. Meanwhile, manually designing loss functions for each specific metric requires expertise and significant manpower. In this paper, we propose to automate the design of metric-specific loss functions by searching differentiable surrogate losses for each metric. We substitute the non-differentiable operations in the metrics with parameterized functions, and conduct parameter search to optimize the shape of loss surfaces. Two constraints are introduced to regularize the search space and make the search efficient. Extensive experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions consistently. The searched losses can generalize well to other datasets and networks. Code shall be released.