# A foundation for systematic analysis of transformers and RNNs for tractography

arXiv: https://arxiv.org/abs/2610.01894 · HTML: https://mlsanity.com/paper/2610.01894/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 70 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [TractSeg - Fast and accurate white matter tract segmentation](https://mlsanity.com/paper/1805.07103/) (2018)
- [Scheduled Sampling for Transformers](https://mlsanity.com/paper/1906.07651/) (2019)
- [Deep Fiber Clustering: Anatomically Informed Unsupervised Deep Learning for Fast and Effective White Matter Parcellation](https://mlsanity.com/paper/2107.04938/) (2021)
- [SuperDTI: Ultrafast diffusion tensor imaging and fiber tractography with deep learning](https://mlsanity.com/paper/2002.01031/) (2020)
- [SupWMA: Consistent and Efficient Tractography Parcellation of Superficial White Matter with Deep Learning](https://mlsanity.com/paper/2201.12528/) (2022)
- [DeepTract: A Probabilistic Deep Learning Framework for White Matter Fiber Tractography](https://mlsanity.com/paper/1812.05129/) (2018)
- [TractoFormer: A Novel Fiber-level Whole Brain Tractography Analysis Framework Using Spectral Embedding and Vision Transformers](https://mlsanity.com/paper/2207.02327/) (2022)
- [FIESTA: Autoencoders for accurate fiber segmentation in tractography](https://mlsanity.com/paper/2212.00143/) (2022)
- [BrainSegNet : A Segmentation Network for Human Brain Fiber Tractography Data into Anatomically Meaningful Clusters](https://mlsanity.com/paper/1710.05158/) (2017)
- [atTRACTive: Semi-automatic white matter tract segmentation using active learning](https://mlsanity.com/paper/2305.18905/) (2023)
- [Modeling the Shape of the Brain Connectome via Deep Neural Networks](https://mlsanity.com/paper/2203.06122/) (2022)
- [BundleSeg: A versatile, reliable and reproducible approach to white matter bundle segmentation](https://mlsanity.com/paper/2308.10958/) (2023)
- [Tract-RLFormer: A Tract-Specific RL policy based Decoder-only Transformer Network](https://mlsanity.com/paper/2411.05757/) (2024)
- [Deep Learning-Based Diffusion MRI Tractography: Integrating Spatial and Anatomical Information](https://mlsanity.com/paper/2503.03329/) (2025)
- [DDTracking: A Deep Generative Framework for Diffusion MRI Tractography with Streamline Local-Global Spatiotemporal Modeling](https://mlsanity.com/paper/2508.04568/) (2025)
- [Streamline tractography of the fetal brain in utero with machine learning](https://mlsanity.com/paper/2408.14326/) (2024)
- [TractoTransformer: Diffusion MRI Streamline Tractography using CNN and Transformer Networks](https://mlsanity.com/paper/2509.16429/) (2025)

## Abstract (arXiv)

Machine learning (ML) has emerged as a promising approach for improving diffusion MRI (dMRI) tractography, a task that remains limited by the intrinsic tension between local diffusion information and global anatomical plausibility. In this work, we systematically evaluate recurrent neural networks (RNNs) and Transformer models for iterative tractography, with particular attention to training strategies, input representations (including convolutional neural network (CNN)-based embeddings and end-of-sequence (EOS) tokens), and hyperparameter selection. We introduce a generation-validation phase enabling supervision at the streamline level during training, allowing supervision despite the mismatch between local loss functions and global streamline quality. Using the ISMRM2015 tractography challenge dataset, our models achieve the highest reported performance to date. Through controlled experiments, we quantify the impact of missing bundles, noisy or imperfect training streamlines, and invalid fibers in the training set. Finally, we demonstrate the applicability of our best-performing models for in vivo data from the Tractoinferno database. Overall, our results highlight both the potential and the limits of sequence-based deep learning models such as Transformers and RNNs for tractography, and emphasize the need for improved phantoms and evaluation methods for in vivo validation. We provide takeaways and recommendations for future researchers training and validating sequence-based supervised methods for tractography.