# SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome

arXiv: https://arxiv.org/abs/2511.15464 · HTML: https://mlsanity.com/paper/2511.15464/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images](https://mlsanity.com/paper/2403.11703/) (2024)
- [Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification](https://mlsanity.com/paper/1504.07947/) (2015)
- [HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis](https://mlsanity.com/paper/2406.16192/) (2024)
- [Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning](https://mlsanity.com/paper/2306.01859/) (2023)
- [Accurate Spatial Gene Expression Prediction by integrating Multi-resolution features](https://mlsanity.com/paper/2403.07592/) (2024)
- [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11618/) (2024)
- [MERGE: Multi-faceted Hierarchical Graph-based GNN for Gene Expression Prediction from Whole Slide Histopathology Images](https://mlsanity.com/paper/2412.02601/) (2024)
- [TopoCellGen: Generating Histopathology Cell Topology with a Diffusion Model](https://mlsanity.com/paper/2412.06011/) (2024)
- [Sparse Structure Learning via Graph Neural Networks for Inductive Document Classification](https://mlsanity.com/paper/2112.06386/) (2021)
- [A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics](https://mlsanity.com/paper/2508.01490/) (2025)
- [Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation](https://mlsanity.com/paper/2504.18856/) (2025)
- [Improving out-of-distribution generalization in graphs via hierarchical semantic environments](https://mlsanity.com/paper/2403.01773/) (2024)
- [SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space](https://mlsanity.com/paper/2506.21857/) (2025)

## Built on by (most cited first, 1 of 1)

- [Can We Go Beyond Visual Features? Neural Tissue Relation Modeling for Relational Graph Analysis in Non-Melanoma Skin Histology](https://mlsanity.com/paper/2512.06949/) (2025)

## Abstract (arXiv)

Recent advances in computational pathology have leveraged vision-language models to learn joint representations of Hematoxylin and Eosin (HE) images with spatial transcriptomic (ST) profiles. However, existing approaches typically align HE tiles with their corresponding ST profiles at a single scale, overlooking fine-grained cellular structures and their spatial organization. To address this, we propose Sigmma, a multi-modal contrastive alignment framework for learning hierarchical representations of HE images and spatial transcriptome profiles across multiple scales. Sigmma introduces multi-scale contrastive alignment, ensuring that representations learned at different scales remain coherent across modalities. Furthermore, by representing cell interactions as a graph and integrating inter- and intra-subgraph relationships, our approach effectively captures cell-cell interactions, ranging from fine to coarse, within the tissue microenvironment. We demonstrate that Sigmm learns representations that better capture cross-modal correspondences, leading to an improvement of avg. 9.78% in the gene-expression prediction task and avg. 26.93% in the cross-modal retrieval task across datasets. We further show that it learns meaningful multi-tissue organization in downstream analyses.