# TractGraphFormer: Anatomically Informed Hybrid Graph CNN-Transformer Network for Interpretable Sex and Age Prediction from Diffusion MRI Tractography

arXiv: https://arxiv.org/abs/2407.08883 · HTML: https://mlsanity.com/paper/2407.08883/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 111 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2211.13202/) (2022)
- [A graph-transformer for whole slide image classification](https://mlsanity.com/paper/2205.09671/) (2022)
- [Tractogram filtering of anatomically non-plausible fibers with geometric deep learning](https://mlsanity.com/paper/2003.11013/) (2020)
- [Model and predict age and sex in healthy subjects using brain white matter features: A deep learning approach](https://mlsanity.com/paper/2202.03595/) (2022)
- [Deep fiber clustering: Anatomically informed fiber clustering with self-supervised deep learning for fast and effective tractography parcellation](https://mlsanity.com/paper/2205.00627/) (2022)
- [TractGeoNet: A geometric deep learning framework for pointwise analysis of tract microstructure to predict language assessment performance](https://mlsanity.com/paper/2307.03982/) (2023)
- [TractCloud: Registration-free tractography parcellation with a novel local-global streamline point cloud representation](https://mlsanity.com/paper/2307.09000/) (2023)
- [White Matter Tracts are Point Clouds: Neuropsychological Score Prediction and Critical Region Localization via Geometric Deep Learning](https://mlsanity.com/paper/2207.02402/) (2022)
- [TractGraphCNN: anatomically informed graph CNN for classification using diffusion MRI tractography](https://mlsanity.com/paper/2301.01911/) (2023)
- [TractoSCR: A Novel Supervised Contrastive Regression Framework for Prediction of Neurocognitive Measures Using Multi-Site Harmonized Diffusion MRI Tractography](https://mlsanity.com/paper/2210.07411/) (2022)
- [RTN: Reinforced Transformer Network for Coronary CT Angiography Vessel-level Image Quality Assessment](https://mlsanity.com/paper/2207.06177/) (2022)
- [TractoFormer: A Novel Fiber-level Whole Brain Tractography Analysis Framework Using Spectral Embedding and Vision Transformers](https://mlsanity.com/paper/2207.02327/) (2022)
- [Reconstructing the somatotopic organization of the corticospinal tract remains a challenge for modern tractography methods](https://mlsanity.com/paper/2306.05623/) (2023)
- [PPA: Principal Parcellation Analysis for Brain Connectomes and Multiple Traits](https://mlsanity.com/paper/2103.03478/) (2021)
- [A Deep Network for Explainable Prediction of Non-Imaging Phenotypes using Anatomical Multi-View Data](https://mlsanity.com/paper/2401.04579/) (2024)
- [Fiber Tract Shape Measures Inform Prediction of Non-Imaging Phenotypes](https://mlsanity.com/paper/2303.09124/) (2023)
- [Cross-domain Fiber Cluster Shape Analysis for Language Performance Cognitive Score Prediction](https://mlsanity.com/paper/2403.19001/) (2024)
- [Predicting age of human subjects based on structural connectivity from diffusion tensor imaging](https://mlsanity.com/paper/1405.5260/) (2014)
- [CASHformer: Cognition Aware SHape Transformer for Longitudinal Analysis](https://mlsanity.com/paper/2207.02091/) (2022)

## Built on by (most cited first, 7 of 7)

- [The shape of the brain's connections is predictive of cognitive performance: an explainable machine learning study](https://mlsanity.com/paper/2410.15108/) (2024)
- [TractCloud-FOV: Deep Learning-based Robust Tractography Parcellation in Diffusion MRI with Incomplete Field of View](https://mlsanity.com/paper/2502.20637/) (2025)
- [White Matter Geometry-Guided Score-Based Diffusion Model for Tissue Microstructure Imputation in Tractography Imaging](https://mlsanity.com/paper/2407.19460/) (2024)
- [A Novel Streamline-based diffusion MRI Tractography Registration Method with Probabilistic Keypoint Detection](https://mlsanity.com/paper/2503.02481/) (2025)
- [Automated Mapping the Pathways of Cranial Nerve II, III, V, and VII/VIII: A Multi-Parametric Multi-Stage Diffusion Tractography Atlas](https://mlsanity.com/paper/2507.23245/) (2025)
- [Tractogram foundation model](https://mlsanity.com/paper/2606.09893/) (2026)
- [FiberGeoText: A Vision-Language Model for Population- Level Organization of Superficial White Matter](https://mlsanity.com/paper/2610.02755/) (2026)

## Abstract (arXiv)

The relationship between brain connections and non-imaging phenotypes is increasingly studied using deep neural networks. However, the local and global properties of brain white matter networks are often overlooked in convolutional network design. We introduce TractGraphFormer, a hybrid Graph CNN-Transformer deep learning framework tailored for diffusion MRI tractography. This model leverages local anatomical characteristics and global feature dependencies of white matter structures. The Graph CNN module captures white matter geometry and grey matter connectivity to aggregate local features from anatomically similar white matter connections, while the Transformer module uses self-attention to enhance global information learning. Additionally, TractGraphFormer includes an attention module for interpreting predictive white matter connections. We apply TractGraphFormer to tasks of sex and age prediction. TractGraphFormer shows strong performance in large datasets of children (n=9345) and young adults (n=1065). Overall, our approach suggests that widespread connections in the WM are predictive of the sex and age of an individual. For each prediction task, consistent predictive anatomical tracts are identified across the two datasets. The proposed approach highlights the potential of integrating local anatomical information and global feature dependencies to improve prediction performance in machine learning with diffusion MRI tractography.