# What makes for good morphology representations for spatial omics?

arXiv: https://arxiv.org/abs/2407.20660 · HTML: https://mlsanity.com/paper/2407.20660/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 88 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning](https://mlsanity.com/paper/2306.01859/) (2023)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Accurate Spatial Gene Expression Prediction by integrating Multi-resolution features](https://mlsanity.com/paper/2403.07592/) (2024)
- [Efficient-CapsNet: Capsule Network with Self-Attention Routing](https://mlsanity.com/paper/2101.12491/) (2021)
- [Predicting molecular phenotypes from histopathology images: a transcriptome-wide expression-morphology analysis in breast cancer](https://mlsanity.com/paper/2009.08917/) (2020)
- [SEPAL: Spatial Gene Expression Prediction from Local Graphs](https://mlsanity.com/paper/2309.01036/) (2023)
- [hist2RNA: An efficient deep learning architecture to predict gene expression from breast cancer histopathology images](https://mlsanity.com/paper/2304.04507/) (2023)
- [Transcriptome-wide prediction of prostate cancer gene expression from histopathology images using co-expression based convolutional neural networks](https://mlsanity.com/paper/2104.09310/) (2021)
- [M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images](https://mlsanity.com/paper/2401.10608/) (2024)
- [Breast Cancer Histopathology Image based Gene Expression Prediction using Spatial Transcriptomics data and Deep Learning](https://mlsanity.com/paper/2303.09987/) (2023)
- [Emerging Artificial Intelligence Applications in Spatial Transcriptomics Analysis](https://mlsanity.com/paper/2203.09664/) (2022)
- [Transcriptome-supervised classification of tissue morphology using deep learning](https://mlsanity.com/paper/2312.04605/) (2023)

## Abstract (arXiv)

Spatial omics has transformed our understanding of tissue architecture by preserving spatial context of gene expression patterns. Simultaneously, advances in imaging AI have enabled extraction of morphological features describing the tissue. The intersection of spatial omics and imaging AI presents opportunities for a more holistic understanding. In this review we introduce a framework for categorizing spatial omics-morphology combination methods, focusing on how morphological features can be translated or integrated into spatial omics analyses. By translation we mean finding morphological features that spatially correlate with gene expression patterns with the purpose of predicting gene expression. Such features can be used to generate super-resolution gene expression maps or infer genetic information from clinical H&E-stained samples. By integration we mean finding morphological features that spatially complement gene expression patterns with the purpose of enriching information. Such features can be used to define spatial domains, especially where gene expression has preceded morphological changes and where morphology remains after gene expression. We discuss learning strategies and directions for further development of the field.