# Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification

arXiv: https://arxiv.org/abs/2303.08446 · HTML: https://mlsanity.com/paper/2303.08446/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Few-shot Image Generation with Elastic Weight Consolidation](https://mlsanity.com/paper/2012.02780/) (2020)
- [Restricting the Flow: Information Bottlenecks for Attribution](https://mlsanity.com/paper/2001.00396/) (2020)
- [An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction](https://mlsanity.com/paper/2005.00652/) (2020)

## Built on by (most cited first, 30 of 54)

- [Attention-Challenging Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2311.07125/) (2023)
- [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11618/) (2024)
- [Multimodal Prototyping for cancer survival prediction](https://mlsanity.com/paper/2407.00224/) (2024)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction](https://mlsanity.com/paper/2401.01646/) (2024)
- [Multistain Pretraining for Slide Representation Learning in Pathology](https://mlsanity.com/paper/2408.02859/) (2024)
- [ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2502.08391/) (2025)
- [Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction](https://mlsanity.com/paper/2402.19326/) (2024)
- [Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosis](https://mlsanity.com/paper/2412.13126/) (2024)
- [Exploring Incompatible Knowledge Transfer in Few-shot Image Generation](https://mlsanity.com/paper/2304.07574/) (2023)
- [Rotation-Agnostic Image Representation Learning for Digital Pathology](https://mlsanity.com/paper/2311.08359/) (2023)
- [Knowledge-enhanced Visual-Language Pretraining for Computational Pathology](https://mlsanity.com/paper/2404.09942/) (2024)
- [Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology](https://mlsanity.com/paper/2409.09369/) (2024)
- [Weakly supervised segmentation with point annotations for histopathology images via contrast-based variational model](https://mlsanity.com/paper/2304.03572/) (2023)
- [Pathology-knowledge Enhanced Multi-instance Prompt Learning for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2407.10814/) (2024)
- [Shapley Values-enabled Progressive Pseudo Bag Augmentation for Whole Slide Image Classification](https://mlsanity.com/paper/2312.05490/) (2023)
- [Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2403.07939/) (2024)
- [Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2410.14195/) (2024)
- [AEM: Attention Entropy Maximization for Multiple Instance Learning based Whole Slide Image Classification](https://mlsanity.com/paper/2406.15303/) (2024)
- [OTSurv: A Novel Multiple Instance Learning Framework for Survival Prediction with Heterogeneity-aware Optimal Transport](https://mlsanity.com/paper/2506.20741/) (2025)
- [From Correlation to Causation: Max-Pooling-Based Multi-Instance Learning Leads to More Robust Whole Slide Image Classification](https://mlsanity.com/paper/2408.09449/) (2024)
- [Less but Better: Parameter-Efficient Fine-Tuning of Large Language Models for Personality Detection](https://mlsanity.com/paper/2504.05411/) (2025)
- [Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation](https://mlsanity.com/paper/2506.18658/) (2025)
- [SC-MIL: Sparsely Coded Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2311.00048/) (2023)
- [Benchmarking PathCLIP for Pathology Image Analysis](https://mlsanity.com/paper/2401.02651/) (2024)
- [Dynamic Graph Information Bottleneck](https://mlsanity.com/paper/2402.06716/) (2024)
- [Prompting Whole Slide Image Based Genetic Biomarker Prediction](https://mlsanity.com/paper/2407.09540/) (2024)
- [Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning](https://mlsanity.com/paper/2603.00667/) (2026)
- [Weakly-Supervised Residual Evidential Learning for Multi-Instance Uncertainty Estimation](https://mlsanity.com/paper/2405.04405/) (2024)
- [Mamba2MIL: State Space Duality Based Multiple Instance Learning for Computational Pathology](https://mlsanity.com/paper/2408.15032/) (2024)

## Abstract (arXiv)

While Multiple Instance Learning (MIL) has shown promising results in digital Pathology Whole Slide Image (WSI) classification, such a paradigm still faces performance and generalization problems due to challenges in high computational costs on Gigapixel WSIs and limited sample size for model training. To deal with the computation problem, most MIL methods utilize a frozen pretrained model from ImageNet to obtain representations first. This process may lose essential information owing to the large domain gap and hinder the generalization of model due to the lack of image-level training-time augmentations. Though Self-supervised Learning (SSL) proposes viable representation learning schemes, the improvement of the downstream task still needs to be further explored in the conversion from the task-agnostic features of SSL to the task-specifics under the partial label supervised learning. To alleviate the dilemma of computation cost and performance, we propose an efficient WSI fine-tuning framework motivated by the Information Bottleneck theory. The theory enables the framework to find the minimal sufficient statistics of WSI, thus supporting us to fine-tune the backbone into a task-specific representation only depending on WSI-level weak labels. The WSI-MIL problem is further analyzed to theoretically deduce our fine-tuning method. Our framework is evaluated on five pathology WSI datasets on various WSI heads. The experimental results of our fine-tuned representations show significant improvements in both accuracy and generalization compared with previous works. Source code will be available at https://github.com/invoker-LL/WSI-finetuning.