# Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classification

arXiv: https://arxiv.org/abs/2210.03664 · HTML: https://mlsanity.com/paper/2210.03664/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis](https://mlsanity.com/paper/1804.06353/) (2018)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Revisiting Multiple Instance Neural Networks](https://mlsanity.com/paper/1610.02501/) (2016)
- [Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images](https://mlsanity.com/paper/2001.01599/) (2020)
- [Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2103.10626/) (2021)
- [Feature-map-level Online Adversarial Knowledge Distillation](https://mlsanity.com/paper/2002.01775/) (2020)
- [A Comprehensive Review of Computer-aided Whole-slide Image Analysis: from Datasets to Feature Extraction, Segmentation, Classification, and Detection Approaches](https://mlsanity.com/paper/2102.10553/) (2021)
- [DGMIL: Distribution Guided Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2206.08861/) (2022)
- [Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imaging](https://mlsanity.com/paper/2111.01556/) (2021)
- [Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis](https://mlsanity.com/paper/2208.08789/) (2022)
- [Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey](https://mlsanity.com/paper/1909.03354/) (2019)

## Built on by (most cited first, 30 of 43)

- [Attention-Challenging Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2311.07125/) (2023)
- [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11618/) (2024)
- [The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification](https://mlsanity.com/paper/2305.17891/) (2023)
- [Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Good Instance Classifier is All You Need](https://mlsanity.com/paper/2307.02249/) (2023)
- [Shapley Values-enabled Progressive Pseudo Bag Augmentation for Whole Slide Image Classification](https://mlsanity.com/paper/2312.05490/) (2023)
- [Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2410.14195/) (2024)
- [AEM: Attention Entropy Maximization for Multiple Instance Learning based Whole Slide Image Classification](https://mlsanity.com/paper/2406.15303/) (2024)
- [Benchmarking Embedding Aggregation Methods in Computational Pathology: A Clinical Data Perspective](https://mlsanity.com/paper/2407.07841/) (2024)
- [FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification](https://mlsanity.com/paper/2409.19720/) (2024)
- [From Correlation to Causation: Max-Pooling-Based Multi-Instance Learning Leads to More Robust Whole Slide Image Classification](https://mlsanity.com/paper/2408.09449/) (2024)
- [Learning from Partial Label Proportions for Whole Slide Image Segmentation](https://mlsanity.com/paper/2405.09041/) (2024)
- [OpenAL: An Efficient Deep Active Learning Framework for Open-Set Pathology Image Classification](https://mlsanity.com/paper/2307.05254/) (2023)
- [Cross-attention-based saliency inference for predicting cancer metastasis on whole slide images](https://mlsanity.com/paper/2309.09412/) (2023)
- [Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning](https://mlsanity.com/paper/2502.21130/) (2025)
- [Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning](https://mlsanity.com/paper/2603.00667/) (2026)
- [Weakly-Supervised Residual Evidential Learning for Multi-Instance Uncertainty Estimation](https://mlsanity.com/paper/2405.04405/) (2024)
- [Hard Negative Sample Mining for Whole Slide Image Classification](https://mlsanity.com/paper/2410.02212/) (2024)
- [RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency](https://mlsanity.com/paper/2411.15076/) (2024)
- [Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignment](https://mlsanity.com/paper/2508.01602/) (2025)
- [PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization](https://mlsanity.com/paper/2503.06482/) (2025)
- [Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification](https://mlsanity.com/paper/2305.05871/) (2023)
- [Beyond Multiple Instance Learning: Full Resolution All-In-Memory End-To-End Pathology Slide Modeling](https://mlsanity.com/paper/2403.04865/) (2024)
- [Continual Multiple Instance Learning with Enhanced Localization for Histopathological Whole Slide Image Analysis](https://mlsanity.com/paper/2507.02395/) (2025)
- [Learning from Majority Label: A Novel Problem in Multi-class Multiple-Instance Learning](https://mlsanity.com/paper/2509.04023/) (2025)
- [ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis](https://mlsanity.com/paper/2509.16900/) (2025)
- [Multi-Scale Heterogeneity-Aware Hypergraph Representation for Histopathology Whole Slide Images](https://mlsanity.com/paper/2404.19334/) (2024)
- [Advancing H&E-to-IHC Stain Translation in Breast Cancer: A Multi-Magnification and Attention-Based Approach](https://mlsanity.com/paper/2408.01929/) (2024)
- [Rethinking Multiple Instance Learning: Developing an Instance-Level Classifier via Weakly-Supervised Self-Training](https://mlsanity.com/paper/2408.04813/) (2024)
- [Exploiting Label-Independent Regularization from Spatial Dependencies for Whole Slide Image Analysis](https://mlsanity.com/paper/2602.19487/) (2026)
- [QG-MIL: A Gated Transformer Aggregator for Domain-Agnostic Multiple Instance Learning in Medical Imaging](https://mlsanity.com/paper/2606.20027/) (2026)

## Abstract (arXiv)

Computer-aided pathology diagnosis based on the classification of Whole Slide Image (WSI) plays an important role in clinical practice, and it is often formulated as a weakly-supervised Multiple Instance Learning (MIL) problem. Existing methods solve this problem from either a bag classification or an instance classification perspective. In this paper, we propose an end-to-end weakly supervised knowledge distillation framework (WENO) for WSI classification, which integrates a bag classifier and an instance classifier in a knowledge distillation framework to mutually improve the performance of both classifiers. Specifically, an attention-based bag classifier is used as the teacher network, which is trained with weak bag labels, and an instance classifier is used as the student network, which is trained using the normalized attention scores obtained from the teacher network as soft pseudo labels for the instances in positive bags. An instance feature extractor is shared between the teacher and the student to further enhance the knowledge exchange between them. In addition, we propose a hard positive instance mining strategy based on the output of the student network to force the teacher network to keep mining hard positive instances. WENO is a plug-and-play framework that can be easily applied to any existing attention-based bag classification methods. Extensive experiments on five datasets demonstrate the efficiency of WENO. Code is available at https://github.com/miccaiif/WENO.