# Transcriptomics-guided Slide Representation Learning in Computational Pathology

arXiv: https://arxiv.org/abs/2405.11618 · HTML: https://mlsanity.com/paper/2405.11618/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 98 references that are in the map.
- First posted on arXiv 19 May 2024.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)

## Built on by (most cited first, 30 of 51)

- [HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis](https://mlsanity.com/paper/2406.16192/) (2024)
- [A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model](https://mlsanity.com/paper/2407.15362/) (2024)
- [PathAlign: A vision-language model for whole slide images in histopathology](https://mlsanity.com/paper/2406.19578/) (2024)
- [Do Multiple Instance Learning Models Transfer?](https://mlsanity.com/paper/2506.09022/) (2025)
- [Multistain Pretraining for Slide Representation Learning in Pathology](https://mlsanity.com/paper/2408.02859/) (2024)
- [Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction](https://mlsanity.com/paper/2411.14001/) (2024)
- [A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Tasks](https://mlsanity.com/paper/2501.15724/) (2025)
- [A Survey of Pathology Foundation Model: Progress and Future Directions](https://mlsanity.com/paper/2504.04045/) (2025)
- [Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning](https://mlsanity.com/paper/2411.13623/) (2024)
- [Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology](https://mlsanity.com/paper/2409.09369/) (2024)
- [PixCell: A generative foundation model for digital histopathology images](https://mlsanity.com/paper/2506.05127/) (2025)
- [Multi-Modal Foundation Models for Computational Pathology: A Survey](https://mlsanity.com/paper/2503.09091/) (2025)
- [PathoHR: Breast Cancer Survival Prediction on High-Resolution Pathological Images](https://mlsanity.com/paper/2503.17970/) (2025)
- [GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathology](https://mlsanity.com/paper/2504.01009/) (2025)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue](https://mlsanity.com/paper/2506.13063/) (2025)
- [Pathology Foundation Models](https://mlsanity.com/paper/2407.21317/) (2024)
- [AI-powered virtual tissues from spatial proteomics for clinical diagnostics and biomedical discovery](https://mlsanity.com/paper/2501.06039/) (2025)
- [TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning](https://mlsanity.com/paper/2512.21331/) (2025)
- [CARE: A Molecular-Guided Foundation Model with Adaptive Region Modeling for Whole Slide Image Analysis](https://mlsanity.com/paper/2602.21637/) (2026)
- [Diagnostic Text-guided Representation Learning in Hierarchical Classification for Pathological Whole Slide Image](https://mlsanity.com/paper/2411.10709/) (2024)
- [RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency](https://mlsanity.com/paper/2411.15076/) (2024)
- [Teaching pathology foundation models to accurately predict gene expression with parameter efficient knowledge transfer](https://mlsanity.com/paper/2504.07061/) (2025)
- [MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention](https://mlsanity.com/paper/2503.00374/) (2025)
- [PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization](https://mlsanity.com/paper/2503.06482/) (2025)
- [SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space](https://mlsanity.com/paper/2506.21857/) (2025)
- [Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment](https://mlsanity.com/paper/2503.09498/) (2025)
- [ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathology](https://mlsanity.com/paper/2503.17564/) (2025)
- [EXAONE Path 2.5: Pathology Foundation Model with Multi-Omics Alignment](https://mlsanity.com/paper/2512.14019/) (2025)
- [Multimodal Learning for Scalable Representation of High-Dimensional Medical Data](https://mlsanity.com/paper/2409.13115/) (2024)

## Abstract (arXiv)

Self-supervised learning (SSL) has been successful in building patch embeddings of small histology images (e.g., 224x224 pixels), but scaling these models to learn slide embeddings from the entirety of giga-pixel whole-slide images (WSIs) remains challenging. Here, we leverage complementary information from gene expression profiles to guide slide representation learning using multimodal pre-training. Expression profiles constitute highly detailed molecular descriptions of a tissue that we hypothesize offer a strong task-agnostic training signal for learning slide embeddings. Our slide and expression (S+E) pre-training strategy, called Tangle, employs modality-specific encoders, the outputs of which are aligned via contrastive learning. Tangle was pre-trained on samples from three different organs: liver (n=6,597 S+E pairs), breast (n=1,020), and lung (n=1,012) from two different species (Homo sapiens and Rattus norvegicus). Across three independent test datasets consisting of 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver WSIs, Tangle shows significantly better few-shot performance compared to supervised and SSL baselines. When assessed using prototype-based classification and slide retrieval, Tangle also shows a substantial performance improvement over all baselines. Code available at https://github.com/mahmoodlab/TANGLE.