# Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratings

arXiv: https://arxiv.org/abs/2205.10355 · HTML: https://mlsanity.com/paper/2205.10355/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 10 of 10)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning](https://mlsanity.com/paper/2003.04696/) (2020)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis](https://mlsanity.com/paper/1904.12894/) (2019)
- [Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth](https://mlsanity.com/paper/1702.03407/) (2017)
- [Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient](https://mlsanity.com/paper/2103.06205/) (2021)
- [Ranger21: a synergistic deep learning optimizer](https://mlsanity.com/paper/2106.13731/) (2021)
- [pymia: A Python package for data handling and evaluation in deep learning-based medical image analysis](https://mlsanity.com/paper/2010.03639/) (2020)
- [Generalized Wasserstein Dice Loss, Test-time Augmentation, and Transformers for the BraTS 2021 challenge](https://mlsanity.com/paper/2112.13054/) (2021)

## Built on by (most cited first, 6 of 6)

- [Approaching Peak Ground Truth](https://mlsanity.com/paper/2301.00243/) (2022)
- [The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting](https://mlsanity.com/paper/2305.08992/) (2023)
- [BrainLesion Suite: A Flexible and User-Friendly Framework for Modular Brain Lesion Image Analysis](https://mlsanity.com/paper/2507.09036/) (2025)
- [QCResUNet: Joint Subject-level and Voxel-level Segmentation Quality Prediction](https://mlsanity.com/paper/2412.07156/) (2024)
- [Training the next generation of physicians for artificial intelligence-assisted clinical neuroradiology: ASNR MICCAI Brain Tumor Segmentation (BraTS) 2025 Lighthouse Challenge education platform](https://mlsanity.com/paper/2509.17281/) (2025)
- [AI Evaluation Should Work With Humans](https://mlsanity.com/paper/2608.13577/) (2026)

## Abstract (arXiv)

Human ratings are abstract representations of segmentation quality. To approximate human quality ratings on scarce expert data, we train surrogate quality estimation models. We evaluate on a complex multi-class segmentation problem, specifically glioma segmentation, following the BraTS annotation protocol. The training data features quality ratings from 15 expert neuroradiologists on a scale ranging from 1 to 6 stars for various computer-generated and manual 3D annotations. Even though the networks operate on 2D images and with scarce training data, we can approximate segmentation quality within a margin of error comparable to human intra-rater reliability. Segmentation quality prediction has broad applications. While an understanding of segmentation quality is imperative for successful clinical translation of automatic segmentation quality algorithms, it can play an essential role in training new segmentation models. Due to the split-second inference times, it can be directly applied within a loss function or as a fully-automatic dataset curation mechanism in a federated learning setting.