# Metrics that matter: Evaluating image quality metrics for medical image generation

arXiv: https://arxiv.org/abs/2505.07175 · HTML: https://mlsanity.com/paper/2505.07175/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 76 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 30 of 37)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Score-based diffusion models for accelerated MRI](https://mlsanity.com/paper/2110.05243/) (2021)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Generating Images with Sparse Representations](https://mlsanity.com/paper/2103.03841/) (2021)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [Effectively Unbiased FID and Inception Score and where to find them](https://mlsanity.com/paper/1911.07023/) (2019)
- [Adaptive Diffusion Priors for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2207.05876/) (2022)
- [CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2303.14081/) (2023)
- [Towards performant and reliable undersampled MR reconstruction via diffusion model sampling](https://mlsanity.com/paper/2203.04292/) (2022)
- [A Non-Parametric Test to Detect Data-Copying in Generative Models](https://mlsanity.com/paper/2004.05675/) (2020)
- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projections](https://mlsanity.com/paper/2106.15427/) (2021)
- [COVID-19 CT Image Synthesis with a Conditional Generative Adversarial Network](https://mlsanity.com/paper/2007.14638/) (2020)
- [Combating Mode Collapse in GAN training: An Empirical Analysis using Hessian Eigenvalues](https://mlsanity.com/paper/2012.09673/) (2020)
- [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://mlsanity.com/paper/2302.04440/) (2023)
- [Feature Extraction for Generative Medical Imaging Evaluation: New Evidence Against an Evolving Trend](https://mlsanity.com/paper/2311.13717/) (2023)
- [Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray images](https://mlsanity.com/paper/2305.07644/) (2023)
- [Similarity and Quality Metrics for MR Image-To-Image Translation](https://mlsanity.com/paper/2405.08431/) (2024)
- [Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician](https://mlsanity.com/paper/2107.04540/) (2021)

## Built on by (most cited first, 12 of 12)

- [MRI Super-Resolution with Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2511.16854/) (2025)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation](https://mlsanity.com/paper/2511.13135/) (2025)
- [One-shot synthesis of rare gastrointestinal lesions improves diagnostic accuracy and clinical training](https://mlsanity.com/paper/2512.24278/) (2025)
- [Evaluating the trustworthiness of the Fr'echet Inception Distance with stochastic embedding representations](https://mlsanity.com/paper/2601.21979/) (2026)
- [CSEval: A Framework for Evaluating Clinical Semantics in Text-to-Image Generation](https://mlsanity.com/paper/2602.12004/) (2026)
- [A Calibrated Memorization Index (MI) for Detecting Training Data Leakage in Generative MRI Models](https://mlsanity.com/paper/2602.13066/) (2026)
- [In Defense of OCTA: The Reconstruction-Utility Gap in OCT-to-OCTA Synthesis](https://mlsanity.com/paper/2608.15626/) (2026)
- [Learning Where and What to Lift for Bi-planar X-ray-to-CT Reconstruction](https://mlsanity.com/paper/2608.17255/) (2026)
- [AnaDiffusion: Anatomically CompositionalLatent Diffusion for Controllable 3D Brain MRI Generation](https://mlsanity.com/paper/2608.23014/) (2026)
- [Destroy Me: Automatic Artifact Generation for Histopathology Images](https://mlsanity.com/paper/2608.27516/) (2026)
- [M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Models](https://mlsanity.com/paper/2609.33769/) (2026)

## Abstract (arXiv)

Evaluating generative models for synthetic medical imaging is crucial yet challenging, especially given the high standards of fidelity, anatomical accuracy, and safety required for clinical applications. Standard evaluation of generated images often relies on no-reference image quality metrics when ground truth images are unavailable, but their reliability in this complex domain is not well established. This study comprehensively assesses commonly used no-reference image quality metrics using brain MRI data, including tumour and vascular images, providing a representative exemplar for the field. We systematically evaluate metric sensitivity to a range of challenges, including noise, distribution shifts, and, critically, localised morphological alterations designed to mimic clinically relevant inaccuracies. We then compare these metric scores against model performance on a relevant downstream segmentation task, analysing results across both controlled image perturbations and outputs from different generative model architectures. Our findings reveal significant limitations: many widely-used no-reference image quality metrics correlate poorly with downstream task suitability and exhibit a profound insensitivity to localised anatomical details crucial for clinical validity. Furthermore, these metrics can yield misleading scores regarding distribution shifts, e.g. data memorisation. This reveals the risk of misjudging model readiness, potentially leading to the deployment of flawed tools that could compromise patient safety. We conclude that ensuring generative models are truly fit for clinical purpose requires a multifaceted validation framework, integrating performance on relevant downstream tasks with the cautious interpretation of carefully selected no-reference image quality metrics.