mlsanityData to 7 Oct 2026

RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency

arXiv 2411.15076 ↗First posted 22 Nov 2024eess.IV cs.CV q-bio.QMcited by 4

Not on the map. Why?

Abstract (arXiv)

Spatial transcriptomics (ST) provides essential spatial context by mapping gene expression within tissue, enabling detailed study of cellular heterogeneity and tissue organization. However, aligning ST data with histology images poses challenges due to inherent spatial distortions and modality-specific variations. Existing methods largely rely on direct alignment, which often fails to capture complex cross-modal relationships. To address these limitations, we propose a novel framework that aligns gene and image features using a ranking-based alignment loss, preserving relative similarity across modalities and enabling robust multi-scale alignment. To further enhance the alignment's stability, we employ self-supervised knowledge distillation with a teacher-student network architecture, which serves as an intra-modal stability regularizer that prevents image-representation drift during cross-modal alignment. Extensive experiments on seven public datasets that encompass gene expression prediction, slide-level classification, and survival analysis demonstrate the efficacy of our method, showing improved alignment and predictive performance over existing methods. Code is available at https://github.com/winston52/RankByGene.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 55 of its 111 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

55 of 111 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  9. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  10. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  11. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  12. 2018HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Imagescited by 387
38 more
  1. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  2. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  3. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 254
  4. 2021UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformercited by 197
  5. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  6. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  7. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  8. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  9. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  10. 2023CellViT: Vision Transformers for Precise Cell Segmentation and Classificationcited by 108
  11. 2024HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysiscited by 104
  12. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  13. 2023Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Imagescited by 91
  14. 2020Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primarycited by 89
  15. 2023Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Predictioncited by 86
  16. 2023Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learningcited by 77
  17. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  18. 2024Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathologycited by 66
  19. 2023Attention-Challenging Multiple Instance Learning for Whole Slide Image Classificationcited by 61
  20. 2024A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Modelcited by 59
  21. 2023Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classificationcited by 58
  22. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  23. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  24. 2024Accurate Spatial Gene Expression Prediction by integrating Multi-resolution featurescited by 52
  25. 2024Transcriptomics-guided Slide Representation Learning in Computational Pathologycited by 50
  26. 2024Multimodal Prototyping for cancer survival predictioncited by 47
  27. 2021Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slidescited by 43
  28. 2022Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classificationcited by 42
  29. 2024Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Predictioncited by 41
  30. 2024Multimodal contrastive learning for spatial gene expression prediction using histology imagescited by 29
  31. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
  32. 2024DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classificationcited by 25
  33. 2021Multi-Class Cell Detection Using Spatial Context Representationcited by 24
  34. 2022Multiple Instance Learning via Iterative Self-Paced Supervised Contrastive Learningcited by 22
  35. 2024STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomicscited by 22
  36. 2024PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentationcited by 14
  37. 2024PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioningcited by 13
  38. 2024M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Imagescited by 13

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learningcited by 3
  2. 2025Gene-DML: Dual-Pathway Multi-Level Discrimination for Gene Expression Prediction from Histopathology Imagescited by 1
  3. 2026Preserving DEG Rankings for Gene Discovery in Histology-Based Spatial Gene Expression Predictioncited by 0
  4. 2026GATE-ST: Gene-Aware Text-image Encoder for Spatial Transcriptomicscited by 0