# SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology

arXiv: https://arxiv.org/abs/2312.15010 · HTML: https://mlsanity.com/paper/2312.15010/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 69 references that are in the map.
- First posted on arXiv 22 Dec 2023.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Promises and Pitfalls of Black-Box Concept Learning Models](https://mlsanity.com/paper/2106.13314/) (2021)
- [Do Concept Bottleneck Models Learn as Intended?](https://mlsanity.com/paper/2105.04289/) (2021)
- [Virchow: A Million-Slide Digital Pathology Foundation Model](https://mlsanity.com/paper/2309.07778/) (2023)
- [A graph-transformer for whole slide image classification](https://mlsanity.com/paper/2205.09671/) (2022)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)
- [Deep linear neural networks with arbitrary loss: All local minima are global](https://mlsanity.com/paper/1712.01473/) (2017)
- [Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathology](https://mlsanity.com/paper/2206.01794/) (2022)
- [Interpretable Neural-Symbolic Concept Reasoning](https://mlsanity.com/paper/2304.14068/) (2023)
- [MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network](https://mlsanity.com/paper/1707.02485/) (2017)
- [VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers](https://mlsanity.com/paper/2203.17247/) (2022)

## Built on by (most cited first, 25 of 25)

- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [ZoomLDM: Latent Diffusion Model for multi-scale image generation](https://mlsanity.com/paper/2411.16969/) (2024)
- [Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance](https://mlsanity.com/paper/2505.11997/) (2025)
- [Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathology](https://mlsanity.com/paper/2501.02922/) (2025)
- [GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathology](https://mlsanity.com/paper/2504.01009/) (2025)
- [Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation](https://mlsanity.com/paper/2504.18856/) (2025)
- [DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction](https://mlsanity.com/paper/2510.00053/) (2025)
- [Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning](https://mlsanity.com/paper/2412.01672/) (2024)
- [RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency](https://mlsanity.com/paper/2411.15076/) (2024)
- [Contrastive Integrated Gradients: A Feature Attribution-Based Method for Explaining Whole Slide Image Classification](https://mlsanity.com/paper/2511.08464/) (2025)
- [Prototype-Based Multiple Instance Learning for Gigapixel Whole Slide Image Classification](https://mlsanity.com/paper/2503.08384/) (2025)
- [MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations](https://mlsanity.com/paper/2602.18792/) (2026)
- [UniSemAlign: Text-Prototype Alignment with a Foundation Encoder for Semi-Supervised Histopathology Segmentation](https://mlsanity.com/paper/2604.09169/) (2026)
- [IPGPhormer: Interpretable Pathology Graph-Transformer for Survival Analysis](https://mlsanity.com/paper/2508.12381/) (2025)
- [A Comprehensive Benchmark of Histopathology Foundation Models for Kidney Digital Pathology Images](https://mlsanity.com/paper/2603.15967/) (2026)
- [Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation](https://mlsanity.com/paper/2605.01144/) (2026)
- [Are Compact Rationales Free? Measuring Tile Selection Headroom in Frozen WSI-MIL](https://mlsanity.com/paper/2605.12575/) (2026)
- [Self-Contrastive Weakly Supervised Learning Framework for Prognostic Prediction Using Whole Slide Images](https://mlsanity.com/paper/2405.15264/) (2024)
- [Hallucination Detection in Virtually-Stained Histology: A Latent Space Baseline](https://mlsanity.com/paper/2411.15060/) (2024)
- [Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing](https://mlsanity.com/paper/2502.00594/) (2025)
- [PA-MIL: Phenotype-Aware Multiple Instance Learning Guided by Language Prompting and Genotype-to-Phenotype Relationships](https://mlsanity.com/paper/2602.02558/) (2026)
- [One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2605.17907/) (2026)
- [Symb-xMIL: Symbolic Explanations for Multiple Instance Learning in Digital Pathology](https://mlsanity.com/paper/2606.06224/) (2026)
- [CoDiR: Confidence-Guided Diffusion Refinement for Semi-Supervised Histopathology Segmentation](https://mlsanity.com/paper/2608.11807/) (2026)
- [Explainable Artificial Intelligence (XAI) in Computational Pathology: Definitions, Taxonomy, and Recommendations](https://mlsanity.com/paper/2608.28820/) (2026)

## Abstract (arXiv)

Introducing interpretability and reasoning into Multiple Instance Learning (MIL) methods for Whole Slide Image (WSI) analysis is challenging, given the complexity of gigapixel slides. Traditionally, MIL interpretability is limited to identifying salient regions deemed pertinent for downstream tasks, offering little insight to the end-user (pathologist) regarding the rationale behind these selections. To address this, we propose Self-Interpretable MIL (SI-MIL), a method intrinsically designed for interpretability from the very outset. SI-MIL employs a deep MIL framework to guide an interpretable branch grounded on handcrafted pathological features, facilitating linear predictions. Beyond identifying salient regions, SI-MIL uniquely provides feature-level interpretations rooted in pathological insights for WSIs. Notably, SI-MIL, with its linear prediction constraints, challenges the prevalent myth of an inevitable trade-off between model interpretability and performance, demonstrating competitive results compared to state-of-the-art methods on WSI-level prediction tasks across three cancer types. In addition, we thoroughly benchmark the local and global-interpretability of SI-MIL in terms of statistical analysis, a domain expert study, and desiderata of interpretability, namely, user-friendliness and faithfulness.