# Self-Supervised Pretraining on Paired Sequences of fMRI Data for Transfer Learning to Brain Decoding Tasks

arXiv: https://arxiv.org/abs/2305.09057 · HTML: https://mlsanity.com/paper/2305.09057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 62 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [BolT: Fused Window Transformers for fMRI Time Series Analysis](https://mlsanity.com/paper/2205.11578/) (2022)
- [Attend and Decode: 4D fMRI Task State Decoding Using Attention Models](https://mlsanity.com/paper/2004.05234/) (2020)
- [Self-Supervised Transformers for fMRI representation](https://mlsanity.com/paper/2112.05761/) (2021)
- [Decoding and mapping task states of the human brain via deep learning](https://mlsanity.com/paper/1801.09858/) (2018)
- [Analyzing Neuroimaging Data Through Recurrent Deep Learning Models](https://mlsanity.com/paper/1810.09945/) (2018)
- [Learning Neural Markers of Schizophrenia Disorder Using Recurrent Neural Networks](https://mlsanity.com/paper/1712.00512/) (2017)

## Abstract (arXiv)

In this work we introduce a self-supervised pretraining framework for transformers on functional Magnetic Resonance Imaging (fMRI) data. First, we pretrain our architecture on two self-supervised tasks simultaneously to teach the model a general understanding of the temporal and spatial dynamics of human auditory cortex during music listening. Our pretraining results are the first to suggest a synergistic effect of multitask training on fMRI data. Second, we finetune the pretrained models and train additional fresh models on a supervised fMRI classification task. We observe significantly improved accuracy on held-out runs with the finetuned models, which demonstrates the ability of our pretraining tasks to facilitate transfer learning. This work contributes to the growing body of literature on transformer architectures for pretraining and transfer learning with fMRI data, and serves as a proof of concept for our pretraining tasks and multitask pretraining on fMRI data.