# Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation

arXiv: https://arxiv.org/abs/2103.12857 · HTML: https://mlsanity.com/paper/2103.12857/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 82 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)
- [Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization](https://mlsanity.com/paper/2009.12829/) (2020)
- [Invariant Representations without Adversarial Training](https://mlsanity.com/paper/1805.09458/) (2018)
- [Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation](https://mlsanity.com/paper/1904.07773/) (2019)
- [A Causal View on Robustness of Neural Networks](https://mlsanity.com/paper/2005.01095/) (2020)
- [Detect and Correct Bias in Multi-Site Neuroimaging Datasets](https://mlsanity.com/paper/2002.05049/) (2020)
- [Domain Adaptation as a Problem of Inference on Graphical Models](https://mlsanity.com/paper/2002.03278/) (2020)
- [Scanner Invariant Representations for Diffusion MRI Harmonization](https://mlsanity.com/paper/1904.05375/) (2019)
- [Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data](https://mlsanity.com/paper/2005.10247/) (2020)
- [GANDALF: Generative Adversarial Networks with Discriminator-Adaptive Loss Fine-tuning for Alzheimer's Disease Diagnosis from MRI](https://mlsanity.com/paper/2008.04396/) (2020)
- [Image-level Harmonization of Multi-Site Data using Image-and-Spatial Transformer Networks](https://mlsanity.com/paper/2006.16741/) (2020)
- [Medical Image Harmonization Using Deep Learning Based Canonical Mapping: Toward Robust and Generalizable Learning in Imaging](https://mlsanity.com/paper/2010.05355/) (2020)

## Built on by (most cited first, 19 of 19)

- [Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection](https://mlsanity.com/paper/2306.09158/) (2023)
- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging: an Experimental Study](https://mlsanity.com/paper/2409.04368/) (2024)
- [Bias in Machine Learning Models Can Be Significantly Mitigated by Careful Training: Evidence from Neuroimaging Studies](https://mlsanity.com/paper/2205.13421/) (2022)
- [HYPO: Hyperspherical Out-of-Distribution Generalization](https://mlsanity.com/paper/2402.07785/) (2024)
- [Applications of Generative Adversarial Networks in Neuroimaging and Clinical Neuroscience](https://mlsanity.com/paper/2206.07081/) (2022)
- [Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation](https://mlsanity.com/paper/2210.17398/) (2022)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [A-Eval: A Benchmark for Cross-Dataset Evaluation of Abdominal Multi-Organ Segmentation](https://mlsanity.com/paper/2309.03906/) (2023)
- [IGUANe: a 3D generalizable CycleGAN for multicenter harmonization of brain MR images](https://mlsanity.com/paper/2402.03227/) (2024)
- [CEmb-SAM: Segment Anything Model with Condition Embedding for Joint Learning from Heterogeneous Datasets](https://mlsanity.com/paper/2308.06957/) (2023)
- [MedShift: identifying shift data for medical dataset curation](https://mlsanity.com/paper/2112.13885/) (2021)
- [Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Space](https://mlsanity.com/paper/2307.13995/) (2023)
- [Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neuroscience](https://mlsanity.com/paper/2308.03175/) (2023)
- [Generative models of MRI-derived neuroimaging features and associated dataset of 18,000 samples](https://mlsanity.com/paper/2407.12897/) (2024)
- [Single Domain Generalization for Alzheimer's Detection from 3D MRIs with Pseudo-Morphological Augmentations and Contrastive Learning](https://mlsanity.com/paper/2505.22465/) (2025)
- [Deep Learning Approaches for Medical Imaging Under Varying Degrees of Label Availability: A Comprehensive Survey](https://mlsanity.com/paper/2504.11588/) (2025)
- [ISLA: A U-Net for MRI-based acute ischemic stroke lesion segmentation with deep supervision, attention, domain adaptation, and ensemble learning](https://mlsanity.com/paper/2601.08732/) (2026)
- [Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors](https://mlsanity.com/paper/2608.18190/) (2026)

## Abstract (arXiv)

Domain shift, the mismatch between training and testing data characteristics, causes significant degradation in the predictive performance in multi-source imaging scenarios. In medical imaging, the heterogeneity of population, scanners and acquisition protocols at different sites presents a significant domain shift challenge and has limited the widespread clinical adoption of machine learning models. Harmonization methods which aim to learn a representation of data invariant to these differences are the prevalent tools to address domain shift, but they typically result in degradation of predictive accuracy. This paper takes a different perspective of the problem: we embrace this disharmony in data and design a simple but effective framework for tackling domain shift. The key idea, based on our theoretical arguments, is to build a pretrained classifier on the source data and adapt this model to new data. The classifier can be fine-tuned for intra-site domain adaptation. We can also tackle situations where we do not have access to ground-truth labels on target data; we show how one can use auxiliary tasks for adaptation; these tasks employ covariates such as age, gender and race which are easy to obtain but nevertheless correlated to the main task. We demonstrate substantial improvements in both intra-site domain adaptation and inter-site domain generalization on large-scale real-world 3D brain MRI datasets for classifying Alzheimer's disease and schizophrenia.