# MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRI

arXiv: https://arxiv.org/abs/2610.02136 · HTML: https://mlsanity.com/paper/2610.02136/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 30 of 38)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study](https://mlsanity.com/paper/2004.03271/) (2020)
- [Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2405.14325/) (2024)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [AnomalyDINO: Boosting Patch-based Few-shot Anomaly Detection with DINOv2](https://mlsanity.com/paper/2405.14529/) (2024)
- [Mask, Stitch, and Re-Sample: Enhancing Robustness and Generalizability in Anomaly Detection through Automatic Diffusion Models](https://mlsanity.com/paper/2305.19643/) (2023)
- [Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI](https://mlsanity.com/paper/2303.03758/) (2023)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Diffusion Models with Implicit Guidance for Medical Anomaly Detection](https://mlsanity.com/paper/2403.08464/) (2024)
- [Unsupervised Pathology Detection: A Deep Dive Into the State of the Art](https://mlsanity.com/paper/2303.00609/) (2023)
- [Challenging Current Semi-Supervised Anomaly Segmentation Methods for Brain MRI](https://mlsanity.com/paper/2109.06023/) (2021)
- [The role of noise in denoising models for anomaly detection in medical images](https://mlsanity.com/paper/2301.08330/) (2023)
- [Guided Reconstruction with Conditioned Diffusion Models for Unsupervised Anomaly Detection in Brain MRIs](https://mlsanity.com/paper/2312.04215/) (2023)
- [Ensembled Cold-Diffusion Restorations for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2407.06635/) (2024)
- [Confidence intervals uncovered: Are we ready for real-world medical imaging AI?](https://mlsanity.com/paper/2409.17763/) (2024)
- [MAD-AD: Masked Diffusion for Unsupervised Brain Anomaly Detection](https://mlsanity.com/paper/2502.16943/) (2025)
- [StRegA: Unsupervised Anomaly Detection in Brain MRIs using a Compact Context-encoding Variational Autoencoder](https://mlsanity.com/paper/2201.13271/) (2022)

## Abstract (arXiv)

Unsupervised anomaly detection (UAD) methods for brain MRI are ranked by a single score, yet that score rests on choices that are rarely reported: how each anomaly map is aligned with the reference, how and on which data the threshold is set, and which false-positive budget, metric, aggregation and lesion definition are used. We present MIRTO, an evaluation protocol that makes these choices explicit and measures their effect. It gates the geometry of every comparison with a registration check and label-free diagnostics of known power, sets thresholds on validation data alone and reports the false-positive volume actually realised on test, repeats each comparison over 15,552 defensible evaluation pipelines, and attaches paired subject-bootstrap intervals with multiplicity control. Applied to four UAD methods trained on the same healthy data and tested on 312 BraTS 2020 subjects, MIRTO showed that an axis-order mismatch between stored maps and the reference lowered a diffusion model's voxel AUROC from 0.873 to 0.583 whilst barely moving its slice-level AUROC. Within each metric, the method explained at least 0.95 of the variance in voxel AUROC and AUPRC and 0.77 in Dice, but only 0.14 in lesion sensitivity, where the lesion definition and hit criterion dominated. A Dice advantage that was significant at validation thresholds vanished at equal realised false-positive burden, and an exact identity attributes it to threshold transfer. A training-free change to REFLECT's latent aggregation raised Dice at equal burden by 0.052. Nine hypotheses were tested against explicit criteria; because the same cohort served to develop the protocol, all inference is exploratory.