# Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union

arXiv: https://arxiv.org/abs/2310.19252 · HTML: https://mlsanity.com/paper/2310.19252/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [SpaceNet: A Remote Sensing Dataset and Challenge Series](https://mlsanity.com/paper/1807.01232/) (2018)
- [Separable Self-attention for Mobile Vision Transformers](https://mlsanity.com/paper/2206.02680/) (2022)
- [Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels](https://mlsanity.com/paper/2302.05666/) (2023)

## Built on by (most cited first, 17 of 17)

- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [ZS-VCOS: Zero-Shot Video Camouflaged Object Segmentation By Optical Flow and Open Vocabulary Object Detection](https://mlsanity.com/paper/2505.01431/) (2025)
- [Cross-Model Transferability of Adversarial Patches in Real-time Segmentation for Autonomous Driving](https://mlsanity.com/paper/2502.16012/) (2025)
- [BS-Mamba for Black-Soil Area Detection On the Qinghai-Tibetan Plateau](https://mlsanity.com/paper/2503.12495/) (2025)
- [SoftPQ: Robust Instance Segmentation Evaluation via Soft Matching and Tunable Thresholds](https://mlsanity.com/paper/2505.12155/) (2025)
- [RankSEG-RMA: An Efficient Segmentation Algorithm via Reciprocal Moment Approximation](https://mlsanity.com/paper/2510.15362/) (2025)
- [HQ-UNet: A Hybrid Quantum-Classical U-Net with a Quantum Bottleneck for Remote Sensing Image Segmentation](https://mlsanity.com/paper/2604.27206/) (2026)
- [PAT: Pixel-wise Adaptive Training for Long-tailed Segmentation](https://mlsanity.com/paper/2404.05393/) (2024)
- [Fine-grained Metrics for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2407.21289/) (2024)
- [Pioneering Precision in Lumbar Spine MRI Segmentation with Advanced Deep Learning and Data Enhancement](https://mlsanity.com/paper/2409.06018/) (2024)
- [Towards Size-invariant Salient Object Detection: A Generic Evaluation and Optimization Approach](https://mlsanity.com/paper/2509.15573/) (2025)
- [MRI Brain Tumor Detection with Computer Vision](https://mlsanity.com/paper/2510.10250/) (2025)
- [PanopMamba: Vision State Space Modeling for Nuclei Panoptic Segmentation](https://mlsanity.com/paper/2601.16631/) (2026)
- [Self-supervised training for high-resolution close-range multispectral remote sensing imagery](https://mlsanity.com/paper/2607.11366/) (2026)
- [A Controlled Evaluation of Model Rankings and Input Reliance in Surface Water Segmentation](https://mlsanity.com/paper/2608.30895/) (2026)
- [Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data](https://mlsanity.com/paper/2609.38165/) (2026)
- [On the Relaxation of Conditional Independence Assumption for Image Segmentation](https://mlsanity.com/paper/2609.38930/) (2026)

## Abstract (arXiv)

Semantic segmentation datasets often exhibit two types of imbalance: textitclass imbalance, where some classes appear more frequently than others and textitsize imbalance, where some objects occupy more pixels than others. This causes traditional evaluation metrics to be biased towards textitmajority classes (e.g. overall pixel-wise accuracy) and textitlarge objects (e.g. mean pixel-wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mIoUs along with corresponding worst-case metrics, thereby offering a more holistic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, we undertake an extensive benchmark study, where we train and evaluate 15 modern neural networks with the proposed metrics on 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead to best practices in optimizing fine-grained metrics. The code is available at hrefhttps://github.com/zifuwanggg/JDTLosseshttps://github.com/zifuwanggg/JDTLosses.