# ST-Align: A Multimodal Foundation Model for Image-Gene Alignment in Spatial Transcriptomics

arXiv: https://arxiv.org/abs/2411.16793 · HTML: https://mlsanity.com/paper/2411.16793/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2024.

## Built on by (most cited first, 12 of 12)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [SToFM: a Multi-scale Foundation Model for Spatial Transcriptomics](https://mlsanity.com/paper/2507.11588/) (2025)
- [Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends](https://mlsanity.com/paper/2504.16134/) (2025)
- [Gene-DML: Dual-Pathway Multi-Level Discrimination for Gene Expression Prediction from Histopathology Images](https://mlsanity.com/paper/2507.14670/) (2025)
- [HiFusion: Hierarchical Intra-Spot Alignment and Regional Context Fusion for Spatial Gene Expression Prediction from Histopathology](https://mlsanity.com/paper/2511.12969/) (2025)
- [Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction Between Feature Alignment and Target Fitting](https://mlsanity.com/paper/2601.18231/) (2026)
- [SPATIA: Multimodal Generation and Prediction of Spatial Cell Phenotypes](https://mlsanity.com/paper/2507.04704/) (2025)
- [HyperST: Hierarchical Hyperbolic Learning for Spatial Transcriptomics Prediction](https://mlsanity.com/paper/2511.22107/) (2025)
- [Spatial Transcriptomics as Images for Large-Scale Pretraining](https://mlsanity.com/paper/2603.13432/) (2026)
- [VISTA: Video Interaction Spatio-Temporal Analysis Benchmark](https://mlsanity.com/paper/2605.01391/) (2026)
- [HierarchicalDAEW: Domain-Aware Edge-Weighted Graph Convolution with Evidential Uncertainty for Multi-Section Spatial Gene Expression Prediction from H&E Histology](https://mlsanity.com/paper/2607.20896/) (2026)

## Abstract (arXiv)

Spatial transcriptomics (ST) provides high-resolution pathological images and whole-transcriptomic expression profiles at individual spots across whole-slide scales. This setting makes it an ideal data source to develop multimodal foundation models. Although recent studies attempted to fine-tune visual encoders with trainable gene encoders based on spot-level, the absence of a wider slide perspective and spatial intrinsic relationships limits their ability to capture ST-specific insights effectively. Here, we introduce ST-Align, the first foundation model designed for ST that deeply aligns image-gene pairs by incorporating spatial context, effectively bridging pathological imaging with genomic features. We design a novel pretraining framework with a three-target alignment strategy for ST-Align, enabling (1) multi-scale alignment across image-gene pairs, capturing both spot- and niche-level contexts for a comprehensive perspective, and (2) cross-level alignment of multimodal insights, connecting localized cellular characteristics and broader tissue architecture. Additionally, ST-Align employs specialized encoders tailored to distinct ST contexts, followed by an Attention-Based Fusion Network (ABFN) for enhanced multimodal fusion, effectively merging domain-shared knowledge with ST-specific insights from both pathological and genomic data. We pre-trained ST-Align on 1.3 million spot-niche pairs and evaluated its performance through two downstream tasks across six datasets, demonstrating superior zero-shot and few-shot capabilities. ST-Align highlights the potential for reducing the cost of ST and providing valuable insights into the distinction of critical compositions within human tissue.