# Five Pitfalls When Assessing Synthetic Medical Images with Reference Metrics

arXiv: https://arxiv.org/abs/2408.06075 · HTML: https://mlsanity.com/paper/2408.06075/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 21 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 4 of 4)

- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [The Brain Tumor Segmentation (BraTS) Challenge 2023: Brain MR Image Synthesis for Tumor Segmentation (BraSyn)](https://mlsanity.com/paper/2305.09011/) (2023)
- [Similarity and Quality Metrics for MR Image-To-Image Translation](https://mlsanity.com/paper/2405.08431/) (2024)
- [Comparison of Image Quality Models for Optimization of Image Processing Systems](https://mlsanity.com/paper/2005.01338/) (2020)

## Built on by (most cited first, 4 of 4)

- [MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment Abilities in MLLMs](https://mlsanity.com/paper/2510.01691/) (2025)
- [Where the Devil Hides: Deepfake Detectors Can No Longer Be Trusted](https://mlsanity.com/paper/2505.08255/) (2025)
- [Style transfer as data augmentation: evaluating unpaired image-to-image translation models in mammography](https://mlsanity.com/paper/2502.02475/) (2025)
- [Feature-Space Guided Diffusion for Realistic Ultrasound Image Synthesis](https://mlsanity.com/paper/2607.11655/) (2026)

## Abstract (arXiv)

Reference metrics have been developed to objectively and quantitatively compare two images. Especially for evaluating the quality of reconstructed or compressed images, these metrics have shown very useful. Extensive tests of such metrics on benchmarks of artificially distorted natural images have revealed which metric best correlate with human perception of quality. Direct transfer of these metrics to the evaluation of generative models in medical imaging, however, can easily lead to pitfalls, because assumptions about image content, image data format and image interpretation are often very different. Also, the correlation of reference metrics and human perception of quality can vary strongly for different kinds of distortions and commonly used metrics, such as SSIM, PSNR and MAE are not the best choice for all situations. We selected five pitfalls that showcase unexpected and probably undesired reference metric scores and discuss strategies to avoid them.