# Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teachers

arXiv: https://arxiv.org/abs/2511.17421 · HTML: https://mlsanity.com/paper/2511.17421/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 36 of its 59 references that are in the map.
- First posted on arXiv 21 Nov 2025.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 30 of 36)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [REPAIR: Removing Representation Bias by Dataset Resampling](https://mlsanity.com/paper/1904.07911/) (2019)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Domain Generalization by Mutual-Information Regularization with Pre-trained Models](https://mlsanity.com/paper/2203.10789/) (2022)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [FairPrune: Achieving Fairness Through Pruning for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2203.02110/) (2022)
- [Improving the Fairness of Chest X-ray Classifiers](https://mlsanity.com/paper/2203.12609/) (2022)
- [Towards Understanding Hierarchical Learning: Benefits of Neural Representations](https://mlsanity.com/paper/2006.13436/) (2020)
- [Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm](https://mlsanity.com/paper/2207.02942/) (2022)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation](https://mlsanity.com/paper/2109.10164/) (2021)
- [TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation](https://mlsanity.com/paper/2202.13393/) (2022)
- [Slicing Through Bias: Explaining Performance Gaps in Medical Image Analysis using Slice Discovery Methods](https://mlsanity.com/paper/2406.12142/) (2024)
- [BiasPruner: Debiased Continual Learning for Medical Image Classification](https://mlsanity.com/paper/2407.08609/) (2024)
- [Confidence estimation of classification based on the distribution of the neural network output layer](https://mlsanity.com/paper/2210.07745/) (2022)

## Built on by (most cited first, 4 of 4)

- [Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients](https://mlsanity.com/paper/2512.23090/) (2025)
- [Mitigating Shortcut Learning via Feature Disentanglement in Medical Imaging: A Benchmark Study](https://mlsanity.com/paper/2602.18502/) (2026)
- [Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP](https://mlsanity.com/paper/2608.12086/) (2026)
- [Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year](https://mlsanity.com/paper/2609.31141/) (2026)

## Abstract (arXiv)

Deep learning models are prone to learning shortcut solutions to problems using spuriously correlated yet irrelevant features of their training data. In high-risk applications such as medical image analysis, this phenomenon may prevent models from using clinically meaningful features when making predictions, potentially leading to poor robustness and harm to patients. We demonstrate that different types of shortcuts (those that are diffuse and spread throughout the image, as well as those that are localized to specific areas) manifest distinctly across network layers and can, therefore, be more effectively targeted through mitigation strategies that target the intermediate layers. We propose a novel knowledge distillation framework that leverages a teacher network fine-tuned on a small subset of task-relevant data to mitigate shortcut learning in a student network trained on a large dataset corrupted with a bias feature. Through extensive experiments on CheXpert, ISIC 2017, and SimBA datasets using various architectures (ResNet-18, AlexNet, DenseNet-121, and 3D CNNs), we demonstrate consistent improvements over traditional Empirical Risk Minimization, augmentation-based bias-mitigation, and group-based bias-mitigation approaches. In many cases, we achieve comparable performance with a baseline model trained on bias-free data, even on out-of-distribution test data. Our results demonstrate the practical applicability of our approach to real-world medical imaging scenarios where bias annotations are limited and shortcut features are difficult to identify a priori.