# Transferring Models Trained on Natural Images to 3D MRI via Position Encoded Slice Models

arXiv: https://arxiv.org/abs/2303.01491 · HTML: https://mlsanity.com/paper/2303.01491/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2101.06871/) (2021)
- [Transfer Learning in Magnetic Resonance Brain Imaging: a Systematic Review](https://mlsanity.com/paper/2102.01530/) (2021)
- [Towards Alzheimer's Disease Classification through Transfer Learning](https://mlsanity.com/paper/1711.11117/) (2017)
- [Improved Brain Age Estimation with Slice-based Set Networks](https://mlsanity.com/paper/2102.04438/) (2021)
- [Brain MRI-based 3D Convolutional Neural Networks for Classification of Schizophrenia and Controls](https://mlsanity.com/paper/2003.08818/) (2020)
- [3D Grid-Attention Networks for Interpretable Age and Alzheimer's Disease Prediction from Structural MRI](https://mlsanity.com/paper/2011.09115/) (2020)
- [MedNet: Pre-trained Convolutional Neural Network Model for the Medical Imaging Tasks](https://mlsanity.com/paper/2110.06512/) (2021)

## Built on by (most cited first, 3 of 3)

- [Enhancing MRI-Based Classification of Alzheimer's Disease with Explainable 3D Hybrid Compact Convolutional Transformers](https://mlsanity.com/paper/2403.16175/) (2024)
- [Efficient Slice Anomaly Detection Network for 3D Brain MRI Volume](https://mlsanity.com/paper/2408.15958/) (2024)
- [FedDAPL: Toward Client-Private Generalization in Federated Learning](https://mlsanity.com/paper/2509.23688/) (2025)

## Abstract (arXiv)

Transfer learning has remarkably improved computer vision. These advances also promise improvements in neuroimaging, where training set sizes are often small. However, various difficulties arise in directly applying models pretrained on natural images to radiologic images, such as MRIs. In particular, a mismatch in the input space (2D images vs. 3D MRIs) restricts the direct transfer of models, often forcing us to consider only a few MRI slices as input. To this end, we leverage the 2D-Slice-CNN architecture of Gupta et al. (2021), which embeds all the MRI slices with 2D encoders (neural networks that take 2D image input) and combines them via permutation-invariant layers. With the insight that the pretrained model can serve as the 2D encoder, we initialize the 2D encoder with ImageNet pretrained weights that outperform those initialized and trained from scratch on two neuroimaging tasks -- brain age prediction on the UK Biobank dataset and Alzheimer's disease detection on the ADNI dataset. Further, we improve the modeling capabilities of 2D-Slice models by incorporating spatial information through position embeddings, which can improve the performance in some cases.