mlsanityData to 14 Oct 2026

Metrics that matter: Evaluating image quality metrics for medical image generation

arXiv 2505.07175 ↗First posted 12 May 2025eess.IV cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Evaluating generative models for synthetic medical imaging is crucial yet challenging, especially given the high standards of fidelity, anatomical accuracy, and safety required for clinical applications. Standard evaluation of generated images often relies on no-reference image quality metrics when ground truth images are unavailable, but their reliability in this complex domain is not well established. This study comprehensively assesses commonly used no-reference image quality metrics using brain MRI data, including tumour and vascular images, providing a representative exemplar for the field. We systematically evaluate metric sensitivity to a range of challenges, including noise, distribution shifts, and, critically, localised morphological alterations designed to mimic clinically relevant inaccuracies. We then compare these metric scores against model performance on a relevant downstream segmentation task, analysing results across both controlled image perturbations and outputs from different generative model architectures. Our findings reveal significant limitations: many widely-used no-reference image quality metrics correlate poorly with downstream task suitability and exhibit a profound insensitivity to localised anatomical details crucial for clinical validity. Furthermore, these metrics can yield misleading scores regarding distribution shifts, e.g. data memorisation. This reveals the risk of misjudging model readiness, potentially leading to the deployment of flawed tools that could compromise patient safety. We conclude that ensuring generative models are truly fit for clinical purpose requires a multifaceted validation framework, integrating performance on relevant downstream tasks with the cautious interpretation of carefully selected no-reference image quality metrics.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

37 of 76 references.

References · 37

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  5. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 521
  6. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 511
  7. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  8. 2021Score-based diffusion models for accelerated MRIcited by 311
  9. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  10. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  11. 2022Diffusion Models for Medical Image Analysis: A Comprehensive Surveycited by 273
  12. 2022Brain Imaging Generation with Latent Diffusion Modelscited by 268
25 more
  1. 2021Generating Images with Sparse Representationscited by 266
  2. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  3. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
  4. 2022Unsupervised Medical Image Translation with Adversarial Diffusion Modelscited by 192
  5. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
  6. 2022Adaptive Diffusion Priors for Accelerated MRI Reconstructioncited by 118
  7. 2023CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesiscited by 65
  8. 2022Towards performant and reliable undersampled MR reconstruction via diffusion model samplingcited by 60
  9. 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
  10. 2023Deep Learning Approaches for Data Augmentation in Medical Imaging: A Reviewcited by 48
  11. 2021Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projectionscited by 33
  12. 2020COVID-19 CT Image Synthesis with a Conditional Generative Adversarial Networkcited by 29
  13. 2020Combating Mode Collapse in GAN training: An Empirical Analysis using Hessian Eigenvaluescited by 27
  14. 2023Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samplescited by 25
  15. 2023Feature Extraction for Generative Medical Imaging Evaluation: New Evidence Against an Evolving Trendcited by 24
  16. 2023Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray imagescited by 23
  17. 2024Similarity and Quality Metrics for MR Image-To-Image Translationcited by 20
  18. 2021Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physiciancited by 16
  19. 2023Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedbackcited by 15
  20. 2022medigan: a Python library of pretrained generative models for medical image synthesiscited by 14
  21. 2025The BIG Argument for AI Safety Casescited by 12
  22. 2023Denoising Diffusion Medical Modelscited by 6
  23. 2023FDDM: Unsupervised Medical Image Translation with a Frequency-Decoupled Diffusion Modelcited by 6
  24. 2023Shape-guided Conditional Latent Diffusion Models for Synthesising Brain Vasculaturecited by 5
  25. 2024Towards Ground-truth-free Evaluation of Any Segmentation in Medical Imagescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025MRI Super-Resolution with Deep Learning: A Comprehensive Surveycited by 7
  2. 2025Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challengescited by 5
  3. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generationcited by 0
  4. 2025One-shot synthesis of rare gastrointestinal lesions improves diagnostic accuracy and clinical trainingcited by 0
  5. 2026Evaluating the trustworthiness of the Fr'echet Inception Distance with stochastic embedding representationscited by 0
  6. 2026CSEval: A Framework for Evaluating Clinical Semantics in Text-to-Image Generationcited by 0
  7. 2026A Calibrated Memorization Index (MI) for Detecting Training Data Leakage in Generative MRI Modelscited by 0
  8. 2026In Defense of OCTA: The Reconstruction-Utility Gap in OCT-to-OCTA Synthesiscited by 0
  9. 2026Learning Where and What to Lift for Bi-planar X-ray-to-CT Reconstructioncited by 0
  10. 2026AnaDiffusion: Anatomically CompositionalLatent Diffusion for Controllable 3D Brain MRI Generationcited by 0
  11. 2026Destroy Me: Automatic Artifact Generation for Histopathology Imagescited by 0
  12. 2026M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Modelscited by 0