# Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning

arXiv: https://arxiv.org/abs/2102.05543 · HTML: https://mlsanity.com/paper/2102.05543/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Domain-Specific Batch Normalization for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1906.03950/) (2019)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)
- [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020)
- [Knowledge Transfer for Melanoma Screening with Deep Learning](https://mlsanity.com/paper/1703.07479/) (2017)

## Built on by (most cited first, 15 of 15)

- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization](https://mlsanity.com/paper/2211.13609/) (2022)
- [Trainable Projected Gradient Method for Robust Fine-tuning](https://mlsanity.com/paper/2303.10720/) (2023)
- [Fast Trainable Projection for Robust Fine-Tuning](https://mlsanity.com/paper/2310.19182/) (2023)
- [ERM++: An Improved Baseline for Domain Generalization](https://mlsanity.com/paper/2304.01973/) (2023)
- [TempT: Temporal consistency for Test-time adaptation](https://mlsanity.com/paper/2303.10536/) (2023)
- [Training BatchNorm Only in Neural Architecture Search and Beyond](https://mlsanity.com/paper/2112.00265/) (2021)
- [Exploring the Transferability of a Foundation Model for Fundus Images: Application to Hypertensive Retinopathy](https://mlsanity.com/paper/2401.15526/) (2024)
- [Multiple Instance Verification](https://mlsanity.com/paper/2407.06544/) (2024)
- [On Fine-Tuned Deep Features for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2210.14083/) (2022)
- [AC-Norm: Effective Tuning for Medical Image Analysis via Affine Collaborative Normalization](https://mlsanity.com/paper/2307.15282/) (2023)
- [One-Shot Domain Incremental Learning](https://mlsanity.com/paper/2403.16707/) (2024)
- [A deep learning model for gastric diffuse-type adenocarcinoma classification in whole slide images](https://mlsanity.com/paper/2104.12478/) (2021)
- [Open Domain Generalization with a Single Network by Regularization Exploiting Pre-trained Features](https://mlsanity.com/paper/2312.05141/) (2023)

## Abstract (arXiv)

Transfer learning from ImageNet is the go-to approach when applying deep learning to medical images. The approach is either to fine-tune a pre-trained model or use it as a feature extractor. Most modern architecture contain batch normalisation layers, and fine-tuning a model with such layers requires taking a few precautions as they consist of trainable and non-trainable weights and have two operating modes: training and inference. Attention is primarily given to the non-trainable weights used during inference, as they are the primary source of unexpected behaviour or degradation in performance during transfer learning. It is typically recommended to fine-tune the model with the batch normalisation layers kept in inference mode during both training and inference. In this paper, we pay closer attention instead to the trainable weights of the batch normalisation layers, and we explore their expressive influence in the context of transfer learning. We find that only fine-tuning the trainable weights (scale and centre) of the batch normalisation layers leads to similar performance as to fine-tuning all of the weights, with the added benefit of faster convergence. We demonstrate this on a variety of seven publicly available medical imaging datasets, using four different model architectures.