# Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

arXiv: https://arxiv.org/abs/2206.13393 · HTML: https://mlsanity.com/paper/2206.13393/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [General Multi-label Image Classification with Transformers](https://mlsanity.com/paper/2011.14027/) (2020)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis](https://mlsanity.com/paper/2008.03483/) (2020)
- [Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain](https://mlsanity.com/paper/2011.04145/) (2020)
- [Tensorizing GAN with High-Order Pooling for Alzheimer's Disease Assessment](https://mlsanity.com/paper/2008.00748/) (2020)
- [Characterization Multimodal Connectivity of Brain Network by Hypergraph GAN for Alzheimer's Disease Analysis](https://mlsanity.com/paper/2107.09953/) (2021)
- [Morphological feature visualization of Alzheimer's disease via Multidirectional Perception GAN](https://mlsanity.com/paper/2111.12886/) (2021)
- [DecGAN: Decoupling Generative Adversarial Network detecting abnormal neural circuits for Alzheimer's disease](https://mlsanity.com/paper/2110.05712/) (2021)

## Built on by (most cited first, 3 of 3)

- [Brain Diffuser: An End-to-End Brain Image to Brain Network Pipeline](https://mlsanity.com/paper/2303.06410/) (2023)
- [Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2302.08641/) (2023)
- [ConnectomeDiffuser: Generative AI Enables Brain Network Construction from Diffusion Tensor Imaging](https://mlsanity.com/paper/2505.22683/) (2025)

## Abstract (arXiv)

Cross-modal fusion of different types of neuroimaging data has shown great promise for predicting the progression of Alzheimer's Disease(AD). However, most existing methods applied in neuroimaging can not efficiently fuse the functional and structural information from multi-modal neuroimages. In this work, a novel cross-modal transformer generative adversarial network(CT-GAN) is proposed to fuse functional information contained in resting-state functional magnetic resonance imaging (rs-fMRI) and structural information contained in Diffusion Tensor Imaging (DTI). The developed bi-attention mechanism can match functional information to structural information efficiently and maximize the capability of extracting complementary information from rs-fMRI and DTI. By capturing the deep complementary information between structural features and functional features, the proposed CT-GAN can detect the AD-related brain connectivity, which could be used as a bio-marker of AD. Experimental results show that the proposed model can not only improve classification performance but also detect the AD-related brain connectivity effectively.