# Multimodal Brain Disease Classification with Functional Interaction Learning from Single fMRI Volume

arXiv: https://arxiv.org/abs/2208.03028 · HTML: https://mlsanity.com/paper/2208.03028/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 5 Aug 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation](https://mlsanity.com/paper/2107.00781/) (2021)
- [Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Disease](https://mlsanity.com/paper/1806.01738/) (2018)
- [Mixed Transformer U-Net For Medical Image Segmentation](https://mlsanity.com/paper/2111.04734/) (2021)
- [Spectral Graph Convolutions for Population-based Disease Prediction](https://mlsanity.com/paper/1703.03020/) (2017)
- [Multi-scale 3D Convolution Network for Video Based Person Re-Identification](https://mlsanity.com/paper/1811.07468/) (2018)
- [Global-Local Transformer for Brain Age Estimation](https://mlsanity.com/paper/2109.01663/) (2021)
- [Multi-modal Graph Learning for Disease Prediction](https://mlsanity.com/paper/2203.05880/) (2022)
- [ASD-DiagNet: A hybrid learning approach for detection of Autism Spectrum Disorder using fMRI data](https://mlsanity.com/paper/1904.07577/) (2019)
- [Edge-variational Graph Convolutional Networks for Uncertainty-aware Disease Prediction](https://mlsanity.com/paper/2009.02759/) (2020)
- [Global-Local Temporal Representations For Video Person Re-Identification](https://mlsanity.com/paper/1908.10049/) (2019)
- [Ensemble learning with 3D convolutional neural networks for connectome-based prediction](https://mlsanity.com/paper/1809.06219/) (2018)

## Built on by (most cited first, 1 of 1)

- [fMRI-TAMCL: Text-Anchored Supervised Multimodal Contrastive Learning for fMRI-Based Brain Disorder Classification](https://mlsanity.com/paper/2610.05880/) (2026)

## Abstract (arXiv)

In neuroimaging analysis, fMRI can well assess the function changes for brain diseases with no obvious structural lesions. To date, most deep-learning-based fMRI studies have employed functional connectivity (FC) as the basic feature for disease classification. However, FC is calculated on time series of predefined regions of interest and neglects detailed information contained in each voxel. Another drawback of using FC is the limited sample size for the training of deep models. The low representation ability of FC leads to poor performance in clinical practice, especially when dealing with multimodal medical data involving multiple types of visual signals and textual records for brain diseases. To overcome this bottleneck problem in the fMRI feature modality, we propose BrainFormer, an end-to-end functional interaction learning method for brain disease classification with single fMRI volume. Unlike traditional deep learning methods that construct convolution and transformers on FC, BrainFormer learns the functional interaction from fMRI signals, by modeling the local cues within each voxel with 3D convolutions and capturing the global correlations among distant regions with specially designed global attention mechanisms from shallow layers to deep layers. Meanwhile, BrainFormer can deal with multimodal medical data including fMRI volume, structural MRI, FC features and phenotypic data to achieve more comprehensive brain disease diagnosis. We evaluate BrainFormer on five independent multi-site datasets on autism, Alzheimer's disease, depression, attention deficit hyperactivity disorder and headache disorders. The results demonstrate its effectiveness and generalizability for multiple brain diseases diagnosis with multimodal features. BrainFormer may promote precision of neuroimaging-based diagnosis in clinical practice and motivate future studies on fMRI analysis.