# Spatio-Temporal Encoding of Brain Dynamics with Surface Masked Autoencoders

arXiv: https://arxiv.org/abs/2308.05474 · HTML: https://mlsanity.com/paper/2308.05474/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention](https://mlsanity.com/paper/2105.13495/) (2021)
- [Spherical U-Net on Cortical Surfaces: Methods and Applications](https://mlsanity.com/paper/1904.00906/) (2019)
- [Surface Vision Transformers: Attention-Based Modelling applied to Cortical Analysis](https://mlsanity.com/paper/2203.16414/) (2022)
- [Improving Phenotype Prediction using Long-Range Spatio-Temporal Dynamics of Functional Connectivity](https://mlsanity.com/paper/2109.03115/) (2021)

## Built on by (most cited first, 3 of 3)

- [Scaling Vision Transformers for Functional MRI with Flat Maps](https://mlsanity.com/paper/2510.13768/) (2025)
- [A Dual-Attention Graph Network for fMRI Data Classification](https://mlsanity.com/paper/2508.13328/) (2025)
- [NeurIPS: Neuro-anatomical Inductive Priors for Sphere-based Brain Decoding](https://mlsanity.com/paper/2605.24993/) (2026)

## Abstract (arXiv)

The development of robust and generalisable models for encoding the spatio-temporal dynamics of human brain activity is crucial for advancing neuroscientific discoveries. However, significant individual variation in the organisation of the human cerebral cortex makes it difficult to identify population-level trends in these signals. Recently, Surface Vision Transformers (SiTs) have emerged as a promising approach for modelling cortical signals, yet they face some limitations in low-data scenarios due to the lack of inductive biases in their architecture. To address these challenges, this paper proposes the surface Masked AutoEncoder (sMAE) and video surface Masked AutoEncoder (vsMAE) - for multivariate and spatio-temporal pre-training of cortical signals over regular icosahedral grids. These models are trained to reconstruct cortical feature maps from masked versions of the input by learning strong latent representations of cortical structure and function. Such representations translate into better modelling of individual phenotypes and enhanced performance in downstream tasks. The proposed approach was evaluated on cortical phenotype regression using data from the young adult Human Connectome Project (HCP) and developing HCP (dHCP). Results show that (v)sMAE pre-trained models improve phenotyping prediction performance on multiple tasks by ≥ 26%, and offer faster convergence relative to models trained from scratch. Finally, we show that pre-training vision transformers on large datasets, such as the UK Biobank (UKB), supports transfer learning to low-data regimes. Our code and pre-trained models are publicly available at https://github.com/metrics-lab/surface-masked-autoencoders .