# Momentum Memory for Knowledge Distillation in Computational Pathology

arXiv: https://arxiv.org/abs/2602.21395 · HTML: https://mlsanity.com/paper/2602.21395/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 24 Feb 2026.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 28 of 28)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks](https://mlsanity.com/paper/2107.13048/) (2021)
- [Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction](https://mlsanity.com/paper/2306.08330/) (2023)
- [Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2403.07719/) (2024)
- [A Survey on Graph-Based Deep Learning for Computational Histopathology](https://mlsanity.com/paper/2107.00272/) (2021)
- [Unsupervised Hyperspectral Image Super-Resolution via Self-Supervised Modality Decoupling](https://mlsanity.com/paper/2412.04802/) (2024)
- [DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2407.03575/) (2024)
- [ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation](https://mlsanity.com/paper/2503.06307/) (2025)
- [Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning](https://mlsanity.com/paper/2512.00818/) (2025)
- [USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval](https://mlsanity.com/paper/2301.06844/) (2023)
- [PointDico: Contrastive 3D Representation Learning Guided by Diffusion Models](https://mlsanity.com/paper/2512.08330/) (2025)
- [ReCo-KD: Region- and Context-Aware Knowledge Distillation for Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2601.08301/) (2026)
- [A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics](https://mlsanity.com/paper/2508.01490/) (2025)
- [Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Images](https://mlsanity.com/paper/2403.10040/) (2024)
- [Computational Pathology for Accurate Prediction of Breast Cancer Recurrence: Development and Validation of a Deep Learning-based Tool](https://mlsanity.com/paper/2409.15491/) (2024)
- [CoCo DistillNet: a Cross-layer Correlation Distillation Network for Pathological Gastric Cancer Segmentation](https://mlsanity.com/paper/2108.12173/) (2021)
- [SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification](https://mlsanity.com/paper/2407.08968/) (2024)
- [Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization](https://mlsanity.com/paper/2508.16479/) (2025)
- [Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains](https://mlsanity.com/paper/2512.22664/) (2025)

## Built on by (most cited first, 4 of 4)

- [PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning](https://mlsanity.com/paper/2603.23957/) (2026)
- [UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservation](https://mlsanity.com/paper/2603.14214/) (2026)
- [Pathway-Structured Privileged Distillation for Deployable Computational Pathology](https://mlsanity.com/paper/2606.02877/) (2026)
- [Can Protein-Derived Knowledge Improve Pathology Foundation Models?](https://mlsanity.com/paper/2609.33178/) (2026)

## Abstract (arXiv)

Multimodal learning that integrates genomics and histopathology has shown strong potential in cancer diagnosis, yet its clinical translation is hindered by the limited availability of paired histology-genomics data. Knowledge distillation (KD) offers a practical solution by transferring genomic supervision into histopathology models, enabling accurate inference using histology alone. However, existing KD methods rely on batch-local alignment, which introduces instability due to limited within-batch comparisons and ultimately degrades performance. To address these limitations, we propose Momentum Memory Knowledge Distillation (MoMKD), a cross-modal distillation framework driven by a momentum-updated memory. This memory aggregates genomic and histopathology information across batches, effectively enlarging the supervisory context available to each mini-batch. Furthermore, we decouple the gradients of the genomics and histology branches, preventing genomic signals from dominating histology feature learning during training and eliminating the modality-gap issue at inference time. Extensive experiments on the TCGA-BRCA benchmark (HER2, PR, and ODX classification tasks) and an independent in-house testing dataset demonstrate that MoMKD consistently outperforms state-of-the-art MIL and multimodal KD baselines, delivering strong performance and generalization under histology-only inference. Overall, MoMKD establishes a robust and generalizable knowledge distillation paradigm for computational pathology.