# Multimodal contrastive learning for spatial gene expression prediction using histology images

arXiv: https://arxiv.org/abs/2407.08216 · HTML: https://mlsanity.com/paper/2407.08216/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 45 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning](https://mlsanity.com/paper/2306.01859/) (2023)
- [stMCDI: Masked Conditional Diffusion Model with Graph Neural Network for Spatial Transcriptomics Data Imputation](https://mlsanity.com/paper/2403.10863/) (2024)

## Built on by (most cited first, 30 of 30)

- [Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Images](https://mlsanity.com/paper/2501.15598/) (2025)
- [Spatially Gene Expression Prediction using Dual-Scale Contrastive Learning](https://mlsanity.com/paper/2506.23827/) (2025)
- [Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics](https://mlsanity.com/paper/2412.00651/) (2024)
- [Img2ST-Net: Efficient High-Resolution Spatial Omics Prediction from Whole Slide Histology Images via Fully Convolutional Image-to-Image Learning](https://mlsanity.com/paper/2508.14393/) (2025)
- [RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency](https://mlsanity.com/paper/2411.15076/) (2024)
- [High-Resolution Spatial Transcriptomics from Histology Images using HisToSGE](https://mlsanity.com/paper/2407.20518/) (2024)
- [Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning](https://mlsanity.com/paper/2506.08854/) (2025)
- [GenAR: Next-Scale Autoregressive Generation for Spatial Gene Expression Prediction](https://mlsanity.com/paper/2510.04315/) (2025)
- [Adapting a Pre-trained Single-Cell Foundation Model to Spatial Gene Expression Generation from Histology Images](https://mlsanity.com/paper/2603.19766/) (2026)
- [A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction](https://mlsanity.com/paper/2604.03630/) (2026)
- [Masked Graph Autoencoders with Contrastive Augmentation for Spatially Resolved Transcriptomics Data](https://mlsanity.com/paper/2408.06377/) (2024)
- [PEaRL: Pathway-Enhanced Representation Learning for Gene and Pathway Expression Prediction from Histology](https://mlsanity.com/paper/2510.03455/) (2025)
- [Dual-Path Knowledge-Augmented Contrastive Alignment Network for Spatially Resolved Transcriptomics](https://mlsanity.com/paper/2511.17685/) (2025)
- [Gene-DML: Dual-Pathway Multi-Level Discrimination for Gene Expression Prediction from Histopathology Images](https://mlsanity.com/paper/2507.14670/) (2025)
- [SCR2-ST: Combine Single Cell with Spatial Transcriptomics for Efficient Active Sampling via Reinforcement Learning](https://mlsanity.com/paper/2512.13635/) (2025)
- [DUET: Dual-Paradigm Adaptive Expert Triage with Single-cell Inductive Prior for Spatial Transcriptomics Prediction](https://mlsanity.com/paper/2605.14104/) (2026)
- [FreqFLD: Towards All-in-One Facial Landmark Detection via Frequency Modulation](https://mlsanity.com/paper/2609.10278/) (2026)
- [Masked adversarial neural network for cell type deconvolution in spatial transcriptomics](https://mlsanity.com/paper/2408.05065/) (2024)
- [Evaluation and Prognostic Validation of Deep Regression Models for WSI-Based Gene-Expression Prediction](https://mlsanity.com/paper/2410.00945/) (2024)
- [SPATIA: Multimodal Generation and Prediction of Spatial Cell Phenotypes](https://mlsanity.com/paper/2507.04704/) (2025)
- [3D-Guided Scalable Flow Matching for Generating Volumetric Tissue Spatial Transcriptomics from Serial Histology](https://mlsanity.com/paper/2511.14613/) (2025)
- [Imaging-anchored multiomics in cardiovascular disease: integrating cardiac imaging, bulk, single-cell, and spatial transcriptomics](https://mlsanity.com/paper/2601.07871/) (2026)
- [Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathology](https://mlsanity.com/paper/2602.13944/) (2026)
- [SpaCRD: Multimodal Deep Fusion of Histology and Spatial Transcriptomics for Cancer Region Detection](https://mlsanity.com/paper/2603.06186/) (2026)
- [CHRep: Cross-modal Histology Representation and Post-hoc Calibration for Spatial Gene Expression Prediction](https://mlsanity.com/paper/2604.21573/) (2026)
- [PaSTel: Anchoring Histology in Spatial Transcriptomics via Multi-Scale Hierarchical Bio-Prior Contrastive Pretraining](https://mlsanity.com/paper/2608.14924/) (2026)
- [BioKERN: Biological Kernel Regularization for Histology-to-Transcriptomics Neighborhood Retrieval](https://mlsanity.com/paper/2608.24823/) (2026)
- [SpaFactor: Lightweight Spatial Context-Aware Gene Program Modeling for Histology-to-Transcriptomics Inference](https://mlsanity.com/paper/2609.28563/) (2026)
- [Towards Scalable Context-Aware Single-Cell Spatial Transcriptomics Prediction from Histology Images](https://mlsanity.com/paper/2609.36429/) (2026)
- [Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology](https://mlsanity.com/paper/2610.05157/) (2026)

## Abstract (arXiv)

In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images (WSIs) stained with Hematoxylin and Eosin (H&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose mclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a "word", integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. Our extensive evaluation of mclSTExp on two breast cancer datasets and a skin squamous cell carcinoma dataset demonstrates its superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.