mlsanityData to 14 Oct 2026

Dataset-Specific Encoding of Image Content Underlies Cross-Dataset Failure in Brain MRI Foundation Models

arXiv 2610.13027 ↗First posted 9 Oct 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Frozen features of brain MRI foundation models lose accuracy on new datasets, a failure usually blamed on the features encoding where each scan came from. We show that the failure lies elsewhere. Across 17 public datasets and eight frozen representations, every representation recognizes the MRI sequence within each dataset, yet most fail on an unseen one. The cause is neither dataset encoding nor a shift of each dataset: the content itself is encoded along different directions in each dataset. This explains which datasets fail, for the sequence and for sex, and why removing the dataset signal with INLP, LEACE, CORAL or adversarial training never improves sequence transfer. Only the broadly pretrained 3DINO and SAM-Med3D encode the sequence consistently enough to transfer; the brain-specific BrainIAC and BrainFM do no better than an untrained network. Code is available at https://github.com/BrainFM/brainfm-profiler/tree/main/representation_utility.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

41 of 65 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Densely Connected Convolutional Networkscited by 6,720
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  8. 2020In Search of Lost Domain Generalizationcited by 1,054
  9. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 830
  10. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  11. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  12. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
29 more
  1. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  2. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  3. 2020Domain Generalization using Causal Matchingcited by 225
  4. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  5. 2023SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Imagescited by 107
  6. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 106
  7. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
  8. 2023The ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge 2023: Intracranial Meningiomacited by 53
  9. 2020Detect and Correct Bias in Multi-Site Neuroimaging Datasetscited by 45
  10. 2019Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effectscited by 43
  11. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 42
  12. 2025Current Pathology Foundation Models are unrobust to Medical Center Differencescited by 39
  13. 2025A generalizable 3D framework and model for self-supervised learning in medical imagingcited by 31
  14. 2024Do Histopathological Foundation Models Eliminate Batch Effects? A Comparative Studycited by 18
  15. 2025Analysis of the MICCAI Brain Tumor Segmentation -- Metastases (BraTS-METS) 2025 Lighthouse Challenge: Brain Metastasis Segmentation on Pre- and Post-treatment MRIcited by 18
  16. 2024Analysis of the BraTS 2023 Intracranial Meningioma Segmentation Challengecited by 15
  17. 2016Exploring the reproducibility of functional connectivity alterations in Parkinson's Diseasecited by 13
  18. 2025A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learningcited by 8
  19. 2025A Modality-agnostic Multi-task Foundation Model for Human Brain Imagingcited by 8
  20. 2024Copycats: the many lives of a publicly available medical imaging datasetcited by 6
  21. 2026Representation Geometry as a Diagnostic for Out-of-Distribution Robustnesscited by 4
  22. 2026A Negative-Control Protocol for Clinical EEG Foundation-Model Benchmarks: Dataset Identity and External-Cohort Stress Testingcited by 4
  23. 2026Batch Effects In Brain Foundation Model Embeddingscited by 3
  24. 2025A Structured Review and Quantitative Profiling of Public Brain MRI Datasets for Foundation Model Developmentcited by 1
  25. 2026Benchmarking the Robustness of Foundation Models for Mammography under Domain Shiftcited by 1
  26. 2026Metadata Supervised Imaging Representations for Modelling and Controlling Acquisition Variabilitycited by 1
  27. 2026Do Medical Foundation Models Generalize on the African Brain?cited by 1
  28. 2026Do 3D Medical Foundation Models See Through MRI Artifacts? A Controlled Study of Representation Robustnesscited by 1
  29. 2026Site Is Decodable Before Pretraining: Negative Controls for Probing Frozen Brain-MRI Foundation Modelscited by 1

Built on by

Not cited yet.