mlsanityData to 7 Oct 2026

Transcriptomics-guided Slide Representation Learning in Computational Pathology

arXiv 2405.11618 ↗First posted 19 May 2024cs.CV cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has been successful in building patch embeddings of small histology images (e.g., 224x224 pixels), but scaling these models to learn slide embeddings from the entirety of giga-pixel whole-slide images (WSIs) remains challenging. Here, we leverage complementary information from gene expression profiles to guide slide representation learning using multimodal pre-training. Expression profiles constitute highly detailed molecular descriptions of a tissue that we hypothesize offer a strong task-agnostic training signal for learning slide embeddings. Our slide and expression (S+E) pre-training strategy, called Tangle, employs modality-specific encoders, the outputs of which are aligned via contrastive learning. Tangle was pre-trained on samples from three different organs: liver (n=6,597 S+E pairs), breast (n=1,020), and lung (n=1,012) from two different species (Homo sapiens and Rattus norvegicus). Across three independent test datasets consisting of 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver WSIs, Tangle shows significantly better few-shot performance compared to supervised and SSL baselines. When assessed using prototype-based classification and slide retrieval, Tangle also shows a substantial performance improvement over all baselines. Code available at https://github.com/mahmoodlab/TANGLE.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 53 of its 98 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

53 of 98 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  5. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  6. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  7. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  8. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  9. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  10. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 254
  13. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  14. 2020Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networkscited by 178
  15. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  16. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  17. 2020Self supervised contrastive learning for digital histopathologycited by 146
  18. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  19. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  20. 2023Artificial Intelligence for Digital and Computational Pathologycited by 106
  21. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  22. 2020Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotationscited by 89
  23. 2023Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Predictioncited by 86
  24. 2023Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learningcited by 77
  25. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  26. 2022Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathologycited by 58
  27. 2023Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classificationcited by 58
  28. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  29. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  30. 2023Histopathology Whole Slide Image Analysis with Heterogeneous Graph Representation Learningcited by 47
  31. 2022Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classificationcited by 42
  32. 2023Pathology-and-genomics Multimodal Transformer for Survival Outcome Predictioncited by 32
  33. 2021Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imagingcited by 31
  34. 2022Giga-SSL: Self-Supervised Learning for Gigapixel Imagescited by 24
  35. 2019Convolutional neural network models for cancer type prediction based on gene expressioncited by 22
  36. 2022Interventional Multi-Instance Learning with Deconfounded Instance-Level Predictioncited by 13
  37. 2023Masked Pre-Training of Transformers for Histology Image Analysiscited by 12
  38. 2022ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Imagescited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 26; 2026: 10.

50 papers.

Sort:

By area:

  1. 2024HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysiscited by 104
  2. 2024A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Modelcited by 59
  3. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 29
  4. 2025Do Multiple Instance Learning Models Transfer?cited by 28
  5. 2024Multistain Pretraining for Slide Representation Learning in Pathologycited by 27
  6. 2024Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Predictioncited by 20
  7. 2024Unsupervised Foundation Model-Agnostic Slide-Level Representation Learningcited by 15
  8. 2025A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Taskscited by 15
  9. 2024Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathologycited by 14
  10. 2025A Survey of Pathology Foundation Model: Progress and Future Directionscited by 14
  11. 2025PixCell: A generative foundation model for digital histopathology imagescited by 11
  12. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
38 more
  1. 2025PathoHR: Breast Cancer Survival Prediction on High-Resolution Pathological Imagescited by 9
  2. 2025GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathologycited by 8
  3. 2025PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialoguecited by 8
  4. 2024Pathology Foundation Modelscited by 7
  5. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 7
  6. 2026CARE: A Molecular-Guided Foundation Model with Adaptive Region Modeling for Whole Slide Image Analysiscited by 6
  7. 2025AI-powered virtual tissues from spatial proteomics for clinical diagnostics and biomedical discoverycited by 5
  8. 2024Diagnostic Text-guided Representation Learning in Hierarchical Classification for Pathological Whole Slide Imagecited by 4
  9. 2024RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistencycited by 4
  10. 2025MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retentioncited by 3
  11. 2025Teaching pathology foundation models to accurately predict gene expression with parameter efficient knowledge transfercited by 3
  12. 2025SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Spacecited by 3
  13. 2025PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantizationcited by 2
  14. 2025Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignmentcited by 2
  15. 2025ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathologycited by 2
  16. 2024Multimodal Learning for Scalable Representation of High-Dimensional Medical Datacited by 1
  17. 2024Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Reviewcited by 1
  18. 2025Towards Spatial Transcriptomics-guided Pathological Image Recognition with Batch-Agnostic Encodercited by 1
  19. 2025Cross-Modality Learning for Predicting IHC Biomarkers from H&E-Stained Whole-Slide Imagescited by 1
  20. 2025Multiple Instance Learning Framework with Masked Hard Instance Mining for Gigapixel Histopathology Image Analysiscited by 1
  21. 2025A Flow Model with Low-Rank Transformers for Incomplete Multimodal Survival Analysiscited by 1
  22. 2025SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptomecited by 1
  23. 2025EXAONE Path 2.5: Pathology Foundation Model with Multi-Omics Alignmentcited by 1
  24. 2026Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histologycited by 1
  25. 2026AGE-MIL: Anchor-Guided Evidence Learning for Patient-Level Predictioncited by 1
  26. 2024Evaluation and Prognostic Validation of Deep Regression Models for WSI-Based Gene-Expression Predictioncited by 0
  27. 2024Multimodal Outer Arithmetic Block Dual Fusion of Whole Slide Images and Omics Data for Precision Oncologycited by 0
  28. 2025Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretrainingcited by 0
  29. 2025Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Frameworkcited by 0
  30. 2025Cross-Stain Contrastive Learning for Paired Immunohistochemistry and Histopathology Slide Representation Learningcited by 0
  31. 2025TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learningcited by 0
  32. 2026Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathologycited by 0
  33. 2026SpaCRD: Multimodal Deep Fusion of Histology and Spatial Transcriptomics for Cancer Region Detectioncited by 0
  34. 2026When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Frameworkcited by 0
  35. 2026JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognosticationcited by 0
  36. 2026Synergistic Information Disentanglement for Omni-modal Slide Representation Learning in Computational Pathologycited by 0
  37. 2026MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attentioncited by 0
  38. 2026Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Gradingcited by 0