# The LUMirage: An independent evaluation of zero-shot performance in the LUMIR challenge

arXiv: https://arxiv.org/abs/2512.15505 · HTML: https://mlsanity.com/paper/2512.15505/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [SynthMorph: learning contrast-invariant registration without acquired images](https://mlsanity.com/paper/2004.10282/) (2020)
- [The iWildCam 2020 Competition Dataset](https://mlsanity.com/paper/2004.10340/) (2020)
- [uniGradICON: A Foundation Model for Medical Image Registration](https://mlsanity.com/paper/2403.05780/) (2024)
- [3D Whole Brain Segmentation using Spatially Localized Atlas Network Tiles](https://mlsanity.com/paper/1903.12152/) (2019)
- [Deep Learning in Medical Image Registration: Magic or Mirage?](https://mlsanity.com/paper/2408.05839/) (2024)
- [Beyond the LUMIR challenge: The pathway to foundational registration models](https://mlsanity.com/paper/2505.24160/) (2025)
- [Mamba? Catch The Hype Or Rethink What Really Helps for Image Registration](https://mlsanity.com/paper/2407.19274/) (2024)
- [UltraCortex: Submillimeter Ultra-High Field 9.4 T Brain MR Image Collection and Manual Cortical Segmentations](https://mlsanity.com/paper/2406.18571/) (2024)
- [Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration](https://mlsanity.com/paper/2406.07361/) (2024)
- [Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies](https://mlsanity.com/paper/2512.01913/) (2025)
- [RealDriveSim: A Realistic Multi-Modal Multi-Task Synthetic Dataset for Autonomous Driving](https://mlsanity.com/paper/2506.16319/) (2025)
- [CLAIRE: Scalable GPU-Accelerated Algorithms for Diffeomorphic Image Registration in 3D](https://mlsanity.com/paper/2401.17493/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Right Prior for the Right Deformation: Rethinking Continuous Deformable Image Registration](https://mlsanity.com/paper/2608.16146/) (2026)
- [Is Deformable Image Registration Ready for Brain Metastasis Reirradiation Dose Accumulation? A Longitudinal MRI Benchmark of Registration Accuracy](https://mlsanity.com/paper/2608.28705/) (2026)

## Abstract (arXiv)

The LUMIR challenge represents an important benchmark for evaluating deformable image registration methods on large-scale neuroimaging data. While the challenge demonstrates that modern deep learning methods achieve competitive accuracy on T1-weighted MRI, it also claims exceptional zero-shot generalization to unseen contrasts and resolutions, assertions that contradict established understanding of domain shift in deep learning. In this paper, we perform an independent re-evaluation of these zero-shot claims using rigorous evaluation protocols while addressing potential sources of instrumentation bias. Our findings reveal a more nuanced picture: (1) deep learning methods perform comparably to iterative optimization on in-distribution T1w images and even on human-adjacent species (macaque), demonstrating improved task understanding; (2) however, performance degrades significantly on out-of-distribution contrasts (T2, T2*, FLAIR), with Cohen's d scores ranging from 0.7-1.5, indicating substantial practical impact on downstream clinical workflows; (3) deep learning methods face scalability limitations on high-resolution data, failing to run on 0.6 mm isotropic images, while iterative methods benefit from increased resolution; and (4) deep methods exhibit high sensitivity to preprocessing choices. These results align with the well-established literature on domain shift and suggest that claims of universal zero-shot superiority require careful scrutiny. We advocate for evaluation protocols that reflect practical clinical and research workflows rather than conditions that may inadvertently favor particular method classes.