# Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection

arXiv: https://arxiv.org/abs/2409.12304 · HTML: https://mlsanity.com/paper/2409.12304/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Community-Aware Transformer for Autism Prediction in fMRI Connectome](https://mlsanity.com/paper/2307.10181/) (2023)
- [Self-Supervised Transformers for fMRI representation](https://mlsanity.com/paper/2112.05761/) (2021)

## Built on by (most cited first, 2 of 2)

- [Region-Aware Reconstruction Strategy for Pre-training fMRI Foundation Model](https://mlsanity.com/paper/2511.00443/) (2025)
- [Learning Robust and Task-Invariant Functional Representation from fMRI through Siamese Self-Supervised Learning](https://mlsanity.com/paper/2605.28990/) (2026)

## Abstract (arXiv)

Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that encompasses a wide variety of symptoms and degrees of impairment, which makes the diagnosis and treatment challenging. Functional magnetic resonance imaging (fMRI) has been extensively used to study brain activity in ASD, and machine learning methods have been applied to analyze resting state fMRI (rs-fMRI) data. However, fewer studies have explored the recent transformer-based models on rs-fMRI data. Given the superiority of transformer models in capturing long-range dependencies in sequence data, we have developed a transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. To address over-fitting in small datasets and enhance the model performance, we propose self-supervised pre-training tasks to reconstruct the randomly masked fMRI time-series data, investigating the effects of various masking strategies. We then finetune the model for the ASD classification task and evaluate it using two public datasets and five-fold cross-validation with different amounts of training data. The experiments show that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step, resulting in an average improvement of 10.8% for AUC and 9.3% for subject accuracy compared with the transformer model trained from scratch across different levels of training data availability. Our code is available on GitHub.