# Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling

arXiv: https://arxiv.org/abs/2408.00311 · HTML: https://mlsanity.com/paper/2408.00311/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Builds on (most cited first, 6 of 6)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm](https://mlsanity.com/paper/1906.03720/) (2019)
- [Multi-modal learning for predicting the genotype of glioma](https://mlsanity.com/paper/2203.10852/) (2022)
- [Learning to Predict RNA Sequence Expressions from Whole Slide Images with Applications for Search and Classification](https://mlsanity.com/paper/2203.13997/) (2022)

## Abstract (arXiv)

In this study, we present a novel approach for predicting genomic information from medical imaging modalities using a transformer-based model. We aim to bridge the gap between imaging and genomics data by leveraging transformer networks, allowing for accurate genomic profile predictions from CT/MRI images. Presently most studies rely on the use of whole slide images (WSI) for the association, which are obtained via invasive methodologies. We propose using only available CT/MRI images to predict genomic sequences. Our transformer based approach is able to efficiently generate associations between multiple sequences based on CT/MRI images alone. This work paves the way for the use of non-invasive imaging modalities for precise and personalized healthcare, allowing for a better understanding of diseases and treatment.