# Cross-Modal Synthesis of Structural MRI and Functional Connectivity Networks via Conditional ViT-GANs

arXiv: https://arxiv.org/abs/2309.08160 · HTML: https://mlsanity.com/paper/2309.08160/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 7 of 7)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up](https://mlsanity.com/paper/2102.07074/) (2021)
- [ViTGAN: Training GANs with Vision Transformers](https://mlsanity.com/paper/2107.04589/) (2021)
- [Unsupervised MRI Reconstruction with Generative Adversarial Networks](https://mlsanity.com/paper/2008.13065/) (2020)
- [MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data](https://mlsanity.com/paper/2211.06726/) (2022)

## Built on by (most cited first, 2 of 2)

- [Generative AI Enables Structural Brain Network Construction from fMRI via Symmetric Diffusion Learning](https://mlsanity.com/paper/2309.16205/) (2023)
- [Multimodal Sheaf-based Network for Glioblastoma Molecular Subtype Prediction](https://mlsanity.com/paper/2508.09717/) (2025)

## Abstract (arXiv)

The cross-modal synthesis between structural magnetic resonance imaging (sMRI) and functional network connectivity (FNC) is a relatively unexplored area in medical imaging, especially with respect to schizophrenia. This study employs conditional Vision Transformer Generative Adversarial Networks (cViT-GANs) to generate FNC data based on sMRI inputs. After training on a comprehensive dataset that included both individuals with schizophrenia and healthy control subjects, our cViT-GAN model effectively synthesized the FNC matrix for each subject, and then formed a group difference FNC matrix, obtaining a Pearson correlation of 0.73 with the actual FNC matrix. In addition, our FNC visualization results demonstrate significant correlations in particular subcortical brain regions, highlighting the model's capability of capturing detailed structural-functional associations. This performance distinguishes our model from conditional CNN-based GAN alternatives such as Pix2Pix. Our research is one of the first attempts to link sMRI and FNC synthesis, setting it apart from other cross-modal studies that concentrate on T1- and T2-weighted MR images or the fusion of MRI and CT scans.