# Tractogram foundation model

arXiv: https://arxiv.org/abs/2606.09893 · HTML: https://mlsanity.com/paper/2606.09893/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 47 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Foundation Models in Radiology: What, How, When, Why and Why Not](https://mlsanity.com/paper/2411.18730/) (2024)
- [Model and predict age and sex in healthy subjects using brain white matter features: A deep learning approach](https://mlsanity.com/paper/2202.03595/) (2022)
- [TractCloud: Registration-free tractography parcellation with a novel local-global streamline point cloud representation](https://mlsanity.com/paper/2307.09000/) (2023)
- [TractGraphCNN: anatomically informed graph CNN for classification using diffusion MRI tractography](https://mlsanity.com/paper/2301.01911/) (2023)
- [TractGraphFormer: Anatomically Informed Hybrid Graph CNN-Transformer Network for Interpretable Sex and Age Prediction from Diffusion MRI Tractography](https://mlsanity.com/paper/2407.08883/) (2024)
- [Predicting age of human subjects based on structural connectivity from diffusion tensor imaging](https://mlsanity.com/paper/1405.5260/) (2014)
- [Predicting Clinical Outcome of Stroke Patients with Tractographic Feature](https://mlsanity.com/paper/1907.10419/) (2019)

## Abstract (arXiv)

Diffusion MRI (dMRI) tractography is the only noninvasive approach for mapping white-matter pathways in the living human brain. It represents each brain as a tractogram: a large, unordered set of three-dimensional streamlines that includes information about both local streamline geometry and whole-brain anatomical organization. This structure makes tractograms a natural but challenging target for representation learning. Existing methods treat streamline classification and subject-level prediction as separate problems: streamline classifiers focus on geometric patterns, whereas subject-level prediction often depends on hand-crafted features. As a result, current methods do not learn reusable representations that connect streamline anatomy with whole-brain inter-subject variation. Here we introduce TractFM, a tractogram foundation model that learns reusable representations directly from whole-brain streamline sets. TractFM combines a local streamline encoder with a permutation-equivariant tractogram encoder, allowing all streamlines from a subject to be contextualized jointly in a single forward pass. Pretraining on dense anatomical tract parcellation, i.e., assigning anatomical labels to individual streamlines, yields two complementary representations: contextualized streamline-level embeddings for tract parcellation and compact subject-level descriptors for downstream prediction of subject phenotypes. Across three tractography algorithms and five dMRI datasets, TractFM transfers to both streamline-level and subject-level tasks. Its frozen representations achieve accurate tract parcellation and predict age and sex across independent datasets. These results show that whole-brain geometric context, learned once, can generalize across tractography pipelines, datasets, and prediction tasks.