Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining
Not on the map. Why?
Abstract (arXiv)
Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator. While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck. This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology. We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains. MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes. It requires no transcriptomics at inference and can be inserted before standard MIL aggregators. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators. MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types: +5.2% on survival prediction, +3.3% on tissue subtyping, and +2.6% on biomarker prediction. Ablations confirm that gene-derived prototypes are the primary source of the gains, while spatial, biological, and pathological analyses show that cross-modal prototype affinities capture spatially coherent molecular programs from H&E alone.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
22 of 44 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2018Attention-based Deep Multiple Instance Learningcited by 1,095
- 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
- 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 555
- 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 478
- 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
- 2024Multimodal Whole Slide Foundation Model for Pathologycited by 162
- 2021BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Imagescited by 132
- 2024HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysiscited by 107
- 2024Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathologycited by 72
- 2024Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathologycited by 67
- 2024Multimodal Prototyping for cancer survival predictioncited by 47
10 more
- 2021Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slidescited by 43
- 2022Handcrafted Histological Transformer (H2T): Unsupervised Representation of Whole Slide Imagescited by 32
- 2024Evidential time-to-event prediction with calibrated uncertainty quantificationcited by 7
- 2025DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Predictioncited by 6
- 2024An evidential time-to-event prediction model based on Gaussian random fuzzy numberscited by 5
- 2026Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learningcited by 4
- 2025SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Spacecited by 3
- 2026Towards Spatial Transcriptomics-driven Pathology Foundation Modelscited by 3
- 2026A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Predictioncited by 3
- 2026MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Modelscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.