# M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Models

arXiv: https://arxiv.org/abs/2609.33769 · HTML: https://mlsanity.com/paper/2609.33769/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm](https://mlsanity.com/paper/1906.03720/) (2019)
- [GANs for Medical Image Synthesis: An Empirical Study](https://mlsanity.com/paper/2105.05318/) (2021)
- [WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis](https://mlsanity.com/paper/2402.19043/) (2024)
- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects](https://mlsanity.com/paper/1910.04597/) (2019)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Feature Extraction for Generative Medical Imaging Evaluation: New Evidence Against an Evolving Trend](https://mlsanity.com/paper/2311.13717/) (2023)
- [Similarity and Quality Metrics for MR Image-To-Image Translation](https://mlsanity.com/paper/2405.08431/) (2024)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [A study of why we need to reassess full reference image quality assessment with medical images](https://mlsanity.com/paper/2405.19097/) (2024)
- [Metrics that matter: Evaluating image quality metrics for medical image generation](https://mlsanity.com/paper/2505.07175/) (2025)
- [Assessing the ability of generative adversarial networks to learn canonical medical image statistics](https://mlsanity.com/paper/2204.12007/) (2022)

## Abstract (arXiv)

Quantitative evaluation of generative models for radiology remains challenging. Clinically relevant structures are often small and infrequent, feature spaces learned from natural images may represent them poorly, and a single summary score cannot distinguish limited fidelity from limited diversity. This study proposes the Medical Multi-axis Maximum Mean Discrepancy score (M3-Score), an evaluation framework based on RadioDINO-s16, a frozen vision transformer pretrained on radiology images. M3-Score reports three complementary axes computed at pre-specified encoder depths: fidelity, measured by an unbiased multi-bandwidth radial basis function (RBF) MMD2 at the final block; memorization, measured by nearest-neighbor distances at 75% depth; and coverage, defined as the fraction of real images with a generated neighbor within their k-nearest-neighbor radius at 33% depth. Reference sets are sampled across subjects to limit the influence of correlated slices. On BraTS brain MRI, the fidelity axis ordered five comparison sets of increasing severity (Spearman ρ = 1.00), and a subject-disjoint real set yielded MMD2 = 0 (permutation p = 1). An unconditional denoising diffusion probabilistic model achieved MMD2 = 0.073 (95% confidence interval [0.071, 0.080]) but covered only 38% of the real distribution. Under progressive mode dropping, k-NN manifold recall increased at all twelve encoder blocks, whereas the proposed coverage estimator decreased monotonically (ρ = -1.00). RadioDINO-s16 features separated real brain MRI from generated samples with a ROC-AUC of 0.819, compared with 0.555 for InceptionV3 and 0.582 for CLIP. Across a twentyfold range of sample sizes, the mean M3 value varied by a factor of 1.05, compared with 2.52 for the Fr'echet Inception Distance.