# Frozen Site-Probe Accuracy Is Not a Standalone Harmonization Score: An Audit of Evaluation Practice for Multi-Site MRI

arXiv: https://arxiv.org/abs/2610.14140 · HTML: https://mlsanity.com/paper/2610.14140/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Automated brain extraction of multi-sequence MRI using artificial neural networks](https://mlsanity.com/paper/1901.11341/) (2019)
- [Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects](https://mlsanity.com/paper/1910.04597/) (2019)
- [HACA3: A Unified Approach for Multi-site MR Image Harmonization](https://mlsanity.com/paper/2212.06065/) (2022)
- [Efficacy of MRI data harmonization in the age of machine learning. A multicenter study across 36 datasets](https://mlsanity.com/paper/2211.04125/) (2022)
- [Disentangled Latent Energy-Based Style Translation: An Image-Level Structural MRI Harmonization Framework](https://mlsanity.com/paper/2402.06875/) (2024)
- [Unpaired Volumetric Harmonization of Brain MRI with Conditional Latent Diffusion](https://mlsanity.com/paper/2408.09315/) (2024)
- [PRISM: Privacy-preserving Inter-Site MRI Harmonization via Disentangled Representation Learning](https://mlsanity.com/paper/2411.06513/) (2024)

## Abstract (arXiv)

Image-level MRI harmonization is often evaluated with a frozen site probe, a site classifier trained on raw scans and applied unchanged to harmonized ones: a drop in its accuracy is read as removal of site effects. We audit this practice on ABIDE I (1,112 T1-weighted scans, 17 sites) with ten output sets from seven harmonization methods and controls for chance, head shape, intensity, preprocessing and target alignment. We find that this verdict depends on how the probe is trained, even among probes that classify raw scans well: across random seeds, its balanced accuracy on the same CycleGAN outputs ranges from 0.13 to 0.59 (chance: 0.06). Low frozen-probe accuracy can also arise without harmonization: a Gaussian blur, which increases the intensity distance to the target site, lowers this accuracy to chance. Moreover, where the frozen probe fails, probes trained on the harmonized scans still recover the site, even from intensity histograms alone: on HACA3 outputs, the frozen probe scores 0.25, a probe trained on those outputs 0.75, and one using only their intensity histograms 0.71. Low frozen-probe accuracy therefore does not show that site information has been removed. We recommend reporting several probes trained to convergence, probes trained on harmonized scans, and target alignment.