mlsanityData to 13 Oct 2026

Approaching Peak Ground Truth

arXiv 2301.00243 ↗First posted 31 Dec 2022cs.LG cs.AI cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Machine learning models are typically evaluated by computing similarity with reference annotations and trained by maximizing similarity with such. Especially in the biomedical domain, annotations are subjective and suffer from low inter- and intra-rater reliability. Since annotations only reflect one interpretation of the real world, this can lead to sub-optimal predictions even though the model achieves high similarity scores. Here, the theoretical concept of PGT is introduced. PGT marks the point beyond which an increase in similarity with the reference annotation stops translating to better RWMP. Additionally, a quantitative technique to approximate PGT by computing inter- and intra-rater reliability is proposed. Finally, four categories of PGT-aware strategies to evaluate and improve model performance are reviewed.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 19 of its 39 references that are in the map. First posted on arXiv 31 Dec 2022.

Builds on

19 of 39 references.

References · 19

  1. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 509
  2. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 507
  3. 2020Are we done with ImageNet?cited by 331
  4. 2018A Probabilistic U-Net for Segmentation of Ambiguous Imagescited by 281
  5. 2018Pros and Cons of GAN Evaluation Measurescited by 224
  6. 2022Metrics reloaded: Recommendations for image analysis validationcited by 192
  7. 2019The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challengecited by 188
  8. 2019A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysiscited by 150
  9. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 123
  10. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
  11. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 91
  12. 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 58
7 more
  1. 2020Computing the Testing Error without a Testing Setcited by 54
  2. 2020An Unsupervised Information-Theoretic Perceptual Quality Metriccited by 28
  3. 2021Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficientcited by 26
  4. 2022Labeling instructions matter in biomedical image analysiscited by 20
  5. 2022Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021cited by 14
  6. 2022Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratingscited by 6
  7. 2022Neural Annotation Refinement: Development of a New 3D Dataset for Adrenal Gland Analysiscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 2.

8 papers.

  1. 2024QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challengecited by 12
  2. 2023Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation mapscited by 11
  3. 2023The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpaintingcited by 8
  4. 2024MultiOrg: A Multi-rater Organoid-detection Datasetcited by 2
  5. 2024Train-Free Segmentation in MRI with Cubical Persistent Homologycited by 1
  6. 2025BrainLesion Suite: A Flexible and User-Friendly Framework for Modular Brain Lesion Image Analysiscited by 1
  7. 2023Framing image registration as a landmark detection problem for label-noise-aware task representation (HitR)cited by 0
  8. 2025Enhancing efficiency in paediatric brain tumour segmentation using a pathologically diverse single-center clinical datasetcited by 0