# Brain Network Transformer

arXiv: https://arxiv.org/abs/2210.06681 · HTML: https://mlsanity.com/paper/2210.06681/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 83 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Graph Neural Networks with Learnable Structural and Positional Representations](https://mlsanity.com/paper/2110.07875/) (2021)
- [Nested Graph Neural Networks](https://mlsanity.com/paper/2110.13197/) (2021)
- [Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention](https://mlsanity.com/paper/2105.13495/) (2021)
- [Differentiable Graph Module (DGM) for Graph Convolutional Networks](https://mlsanity.com/paper/2002.04999/) (2020)
- [BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networks](https://mlsanity.com/paper/2204.07054/) (2022)
- [GRPE: Relative Positional Encoding for Graph Transformer](https://mlsanity.com/paper/2201.12787/) (2022)
- [Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future](https://mlsanity.com/paper/2105.13137/) (2021)
- [FBNETGEN: Task-aware GNN-based fMRI Analysis via Functional Brain Network Generation](https://mlsanity.com/paper/2205.12465/) (2022)
- [Gophormer: Ego-Graph Transformer for Node Classification](https://mlsanity.com/paper/2110.13094/) (2021)
- [Interpretable Graph Neural Networks for Connectome-Based Brain Disorder Analysis](https://mlsanity.com/paper/2207.00813/) (2022)
- [On Positional and Structural Node Features for Graph Neural Networks on Non-attributed Graphs](https://mlsanity.com/paper/2107.01495/) (2021)
- [Graph Neural Network for Interpreting Task-fMRI Biomarkers](https://mlsanity.com/paper/1907.01661/) (2019)
- [Joint Embedding of Structural and Functional Brain Networks with Graph Neural Networks for Mental Illness Diagnosis](https://mlsanity.com/paper/2107.03220/) (2021)
- [Adaptive Kernel Graph Neural Network](https://mlsanity.com/paper/2112.04575/) (2021)
- [Data-Efficient Brain Connectome Analysis via Multi-Task Meta-Learning](https://mlsanity.com/paper/2206.04486/) (2022)

## Built on by (most cited first, 30 of 94)

- [Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking](https://mlsanity.com/paper/2409.19407/) (2024)
- [QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training](https://mlsanity.com/paper/2506.00711/) (2025)
- [SwiFT: Swin 4D fMRI Transformer](https://mlsanity.com/paper/2307.05916/) (2023)
- [NeuroGraph: Benchmarks for Graph Machine Learning in Brain Connectomics](https://mlsanity.com/paper/2306.06202/) (2023)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [Community-Aware Transformer for Autism Prediction in fMRI Connectome](https://mlsanity.com/paper/2307.10181/) (2023)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [Population Transformer: Learning Population-level Representations of Neural Activity](https://mlsanity.com/paper/2406.03044/) (2024)
- [Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networks](https://mlsanity.com/paper/2308.11825/) (2023)
- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [Towards a general-purpose foundation model for fMRI analysis](https://mlsanity.com/paper/2506.11167/) (2025)
- [MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding](https://mlsanity.com/paper/2502.15786/) (2025)
- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [Learning Task-Aware Effective Brain Connectivity for fMRI Analysis with Graph Neural Networks](https://mlsanity.com/paper/2211.00261/) (2022)
- [RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inference](https://mlsanity.com/paper/2302.02292/) (2023)
- [Long-range Brain Graph Transformer](https://mlsanity.com/paper/2501.01100/) (2025)
- [PTGB: Pre-Train Graph Neural Networks for Brain Network Analysis](https://mlsanity.com/paper/2305.14376/) (2023)
- [Learning High-Order Relationships of Brain Regions](https://mlsanity.com/paper/2312.02203/) (2023)
- [Biologically Plausible Brain Graph Transformer](https://mlsanity.com/paper/2502.08958/) (2025)
- [BrainNPT: Pre-training of Transformer networks for brain network classification](https://mlsanity.com/paper/2305.01666/) (2023)
- [R-Mixup: Riemannian Mixup for Biological Networks](https://mlsanity.com/paper/2306.02532/) (2023)
- [Multi-modal Cross-domain Self-supervised Pre-training for fMRI and EEG Fusion](https://mlsanity.com/paper/2409.19130/) (2024)
- [Rethinking Functional Brain Connectome Analysis: Do Graph Deep Learning Models Help](https://mlsanity.com/paper/2501.17207/) (2025)
- [Transformer-Based Hierarchical Clustering for Brain Network Analysis](https://mlsanity.com/paper/2305.04142/) (2023)
- [Dynamic Brain Transformer with Multi-level Attention for Functional Brain Network Analysis](https://mlsanity.com/paper/2309.01941/) (2023)
- [MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning](https://mlsanity.com/paper/2312.14574/) (2023)
- [fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding](https://mlsanity.com/paper/2511.21760/) (2025)
- [QuMoS: A Framework for Preserving Security of Quantum Machine Learning Model](https://mlsanity.com/paper/2304.11511/) (2023)
- [Beyond the Snapshot: Brain Tokenized Graph Transformer for Longitudinal Brain Functional Connectome Embedding](https://mlsanity.com/paper/2307.00858/) (2023)
- [BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning](https://mlsanity.com/paper/2405.18808/) (2024)

## Abstract (arXiv)

Human brains are commonly modeled as networks of Regions of Interest (ROIs) and their connections for the understanding of brain functions and mental disorders. Recently, Transformer-based models have been studied over different types of data, including graphs, shown to bring performance gains widely. In this work, we study Transformer-based models for brain network analysis. Driven by the unique properties of data, we model brain networks as graphs with nodes of fixed size and order, which allows us to (1) use connection profiles as node features to provide natural and low-cost positional information and (2) learn pair-wise connection strengths among ROIs with efficient attention weights across individuals that are predictive towards downstream analysis tasks. Moreover, we propose an Orthonormal Clustering Readout operation based on self-supervised soft clustering and orthonormal projection. This design accounts for the underlying functional modules that determine similar behaviors among groups of ROIs, leading to distinguishable cluster-aware node embeddings and informative graph embeddings. Finally, we re-standardize the evaluation pipeline on the only one publicly available large-scale brain network dataset of ABIDE, to enable meaningful comparison of different models. Experiment results show clear improvements of our proposed Brain Network Transformer on both the public ABIDE and our restricted ABCD datasets. The implementation is available at https://github.com/Wayfear/BrainNetworkTransformer.