# Spectral decoupling allows training transferable neural networks in medical imaging

arXiv: https://arxiv.org/abs/2103.17171 · HTML: https://mlsanity.com/paper/2103.17171/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Can AI help in screening Viral and COVID-19 pneumonia?](https://mlsanity.com/paper/2003.13145/) (2020)
- [Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology](https://mlsanity.com/paper/1902.06543/) (2019)
- [COVID-19 Image Data Collection: Prospective Predictions Are the Future](https://mlsanity.com/paper/2006.11988/) (2020)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [UniformAugment: A Search-free Probabilistic Data Augmentation Approach](https://mlsanity.com/paper/2003.14348/) (2020)
- [Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard](https://mlsanity.com/paper/1808.05883/) (2018)
- [On the Learning Dynamics of Deep Neural Networks](https://mlsanity.com/paper/1809.06848/) (2018)

## Built on by (most cited first, 4 of 4)

- [Augment like there's no tomorrow: Consistently performing neural networks for medical imaging](https://mlsanity.com/paper/2206.15274/) (2022)
- [INSIGHT: Explainable Weakly-Supervised Medical Image Analysis](https://mlsanity.com/paper/2412.02012/) (2024)
- [A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images](https://mlsanity.com/paper/2502.19217/) (2025)
- [Optimized Deep Learning Models for Malware Detection under Concept Drift](https://mlsanity.com/paper/2308.10821/) (2023)

## Abstract (arXiv)

Many current neural networks for medical imaging generalise poorly to data unseen during training. Such behaviour can be caused by networks overfitting easy-to-learn, or statistically dominant, features while disregarding other potentially informative features. For example, indistinguishable differences in the sharpness of the images from two different scanners can degrade the performance of the network significantly. All neural networks intended for clinical practice need to be robust to variation in data caused by differences in imaging equipment, sample preparation and patient populations. To address these challenges, we evaluate the utility of spectral decoupling as an implicit bias mitigation method. Spectral decoupling encourages the neural network to learn more features by simply regularising the networks' unnormalised prediction scores with an L2 penalty, thus having no added computational costs. We show that spectral decoupling allows training neural networks on datasets with strong spurious correlations and increases networks' robustness for data distribution shifts. To validate our findings, we train networks with and without spectral decoupling to detect prostate cancer tissue slides and COVID-19 in chest radiographs. Networks trained with spectral decoupling achieve up to 9.5 percent point higher performance on external datasets. Our results show that spectral decoupling helps with generalisation issues associated with neural networks, and can be used to complement or replace computationally expensive explicit bias mitigation methods, such as stain normalization in histological images. We recommend using spectral decoupling as an implicit bias mitigation method in any neural network intended for clinical use.