# Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness

arXiv: https://arxiv.org/abs/2510.27213 · HTML: https://mlsanity.com/paper/2510.27213/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 46 of its 76 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 30 of 46)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [Rethinking Experience Replay: a Bag of Tricks for Continual Learning](https://mlsanity.com/paper/2010.05595/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/2507.10485/) (2025)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [A Comprehensive Survey on Knowledge Distillation](https://mlsanity.com/paper/2503.12067/) (2025)
- [The Challenges of Continuous Self-Supervised Learning](https://mlsanity.com/paper/2203.12710/) (2022)
- [Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning](https://mlsanity.com/paper/2311.17597/) (2023)
- [How Well Does Self-Supervised Pre-Training Perform with Streaming Data?](https://mlsanity.com/paper/2104.12081/) (2021)
- [Error Sensitivity Modulation based Experience Replay: Mitigating Abrupt Representation Drift in Continual Learning](https://mlsanity.com/paper/2302.11344/) (2023)
- [Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation](https://mlsanity.com/paper/2412.08139/) (2024)
- [CDDSA: Contrastive Domain Disentanglement and Style Augmentation for Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2211.12081/) (2022)
- [Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification](https://mlsanity.com/paper/2104.13298/) (2021)
- [Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imaging](https://mlsanity.com/paper/2308.06534/) (2023)
- [Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images](https://mlsanity.com/paper/2206.03009/) (2022)
- [Self-Supervised Learning for Gastritis Detection with Gastric X-ray Images](https://mlsanity.com/paper/2104.02864/) (2021)
- [Incremental Learning for Heterogeneous Structure Segmentation in Brain Tumor MRI](https://mlsanity.com/paper/2305.19404/) (2023)
- [TriBYOL: Triplet BYOL for Self-Supervised Representation Learning](https://mlsanity.com/paper/2206.03012/) (2022)

## Built on by (most cited first, 1 of 1)

- [InvCoSS: Inversion-driven Continual Self-supervised Learning in Medical Multi-modal Image Pre-training](https://mlsanity.com/paper/2512.19213/) (2025)

## Abstract (arXiv)

We propose a novel continual self-supervised learning (CSSL) framework for simultaneously learning diverse features from multi-window-obtained chest computed tomography (CT) images and ensuring data privacy. Achieving a robust and highly generalizable model in medical image diagnosis is challenging, mainly because of issues, such as the scarcity of large-scale, accurately annotated datasets and domain shifts inherent to dynamic healthcare environments. Specifically, in chest CT, these domain shifts often arise from differences in window settings, which are optimized for distinct clinical purposes. Previous CSSL frameworks often mitigated domain shift by reusing past data, a typically impractical approach owing to privacy constraints. Our approach addresses these challenges by effectively capturing the relationship between previously learned knowledge and new information across different training stages through continual pretraining on unlabeled images. Specifically, by incorporating a latent replay-based mechanism into CSSL, our method mitigates catastrophic forgetting due to domain shifts during continual pretraining while ensuring data privacy. Additionally, we introduce a feature distillation technique that integrates Wasserstein distance-based knowledge distillation (WKD) and batch-knowledge ensemble (BKE), enhancing the ability of the model to learn meaningful, domain-shift-robust representations. Finally, we validate our approach using chest CT images obtained across two different window settings, demonstrating superior performance compared with other approaches.