# Self-Supervised Transformers for fMRI representation

arXiv: https://arxiv.org/abs/2112.05761 · HTML: https://mlsanity.com/paper/2112.05761/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 10 Dec 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [Spatio-Temporal Graph Convolution for Resting-State fMRI Analysis](https://mlsanity.com/paper/2003.10613/) (2020)
- [Learning Neural Markers of Schizophrenia Disorder Using Recurrent Neural Networks](https://mlsanity.com/paper/1712.00512/) (2017)

## Built on by (most cited first, 17 of 17)

- [SwiFT: Swin 4D fMRI Transformer](https://mlsanity.com/paper/2307.05916/) (2023)
- [WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series](https://mlsanity.com/paper/2203.09978/) (2022)
- [Towards a general-purpose foundation model for fMRI analysis](https://mlsanity.com/paper/2506.11167/) (2025)
- [DreaMR: Diffusion-driven Counterfactual Explanation for Functional MRI](https://mlsanity.com/paper/2307.09547/) (2023)
- [DSAM: A Deep Learning Framework for Analyzing Temporal and Spatial Dynamics in Brain Networks](https://mlsanity.com/paper/2405.15805/) (2024)
- [Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data](https://mlsanity.com/paper/2509.18507/) (2025)
- [See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI](https://mlsanity.com/paper/2403.06361/) (2024)
- [Scaling Vision Transformers for Functional MRI with Flat Maps](https://mlsanity.com/paper/2510.13768/) (2025)
- [Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection](https://mlsanity.com/paper/2409.12304/) (2024)
- [BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data](https://mlsanity.com/paper/2506.22591/) (2025)
- [VarCoNet: A variability-aware self-supervised framework for functional connectome extraction from resting-state fMRI](https://mlsanity.com/paper/2510.02120/) (2025)
- [Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?](https://mlsanity.com/paper/2604.03619/) (2026)
- [Self-Supervised Pretraining on Paired Sequences of fMRI Data for Transfer Learning to Brain Decoding Tasks](https://mlsanity.com/paper/2305.09057/) (2023)
- [Sequential Transfer Learning to Decode Heard and Imagined Timbre from fMRI Data](https://mlsanity.com/paper/2305.13226/) (2023)
- [SpaRG: Sparsely Reconstructed Graphs for Generalizable fMRI Analysis](https://mlsanity.com/paper/2410.07201/) (2024)
- [Atlas-free Brain Network Transformer](https://mlsanity.com/paper/2510.03306/) (2025)
- [Adapting HFMCA to Graph Data: Self-Supervised Learning for Generalizable fMRI Representations](https://mlsanity.com/paper/2510.05177/) (2025)

## Abstract (arXiv)

We present TFF, which is a Transformer framework for the analysis of functional Magnetic Resonance Imaging (fMRI) data. TFF employs a two-phase training approach. First, self-supervised training is applied to a collection of fMRI scans, where the model is trained to reconstruct 3D volume data. Second, the pre-trained model is fine-tuned on specific tasks, utilizing ground truth labels. Our results show state-of-the-art performance on a variety of fMRI tasks, including age and gender prediction, as well as schizophrenia recognition. Our code for the training, network architecture, and results is attached as supplementary material.