# SwiFT: Swin 4D fMRI Transformer

arXiv: https://arxiv.org/abs/2307.05916 · HTML: https://mlsanity.com/paper/2307.05916/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2207.10866/) (2022)
- [Brain Network Transformer](https://mlsanity.com/paper/2210.06681/) (2022)
- [Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention](https://mlsanity.com/paper/2105.13495/) (2021)
- [Spatio-Temporal Graph Convolution for Resting-State fMRI Analysis](https://mlsanity.com/paper/2003.10613/) (2020)
- [Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values](https://mlsanity.com/paper/1810.03307/) (2018)
- [BolT: Fused Window Transformers for fMRI Time Series Analysis](https://mlsanity.com/paper/2205.11578/) (2022)
- [Attend and Decode: 4D fMRI Task State Decoding Using Attention Models](https://mlsanity.com/paper/2004.05234/) (2020)
- [Self-Supervised Transformers for fMRI representation](https://mlsanity.com/paper/2112.05761/) (2021)
- [Challenges for machine learning in clinical translation of big data imaging studies](https://mlsanity.com/paper/2107.05630/) (2021)

## Built on by (most cited first, 30 of 36)

- [Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking](https://mlsanity.com/paper/2409.19407/) (2024)
- [NeuralBench: A Unifying Framework to Benchmark NeuroAI Models](https://mlsanity.com/paper/2605.08495/) (2026)
- [Towards a general-purpose foundation model for fMRI analysis](https://mlsanity.com/paper/2506.11167/) (2025)
- [SLIM-Brain: A Data- and Training-Efficient Foundation Model for fMRI Data Analysis](https://mlsanity.com/paper/2512.21881/) (2025)
- [Hierarchical Quantum Control Gates for Functional MRI Understanding](https://mlsanity.com/paper/2408.03596/) (2024)
- [GyroSwin: 5D Surrogates for Gyrokinetic Plasma Turbulence Simulations](https://mlsanity.com/paper/2510.07314/) (2025)
- [fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding](https://mlsanity.com/paper/2511.21760/) (2025)
- [BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning](https://mlsanity.com/paper/2405.18808/) (2024)
- [Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis](https://mlsanity.com/paper/2408.13074/) (2024)
- [FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI](https://mlsanity.com/paper/2606.11500/) (2026)
- [Voxel-Level Brain States Prediction Using Swin Transformer](https://mlsanity.com/paper/2506.11455/) (2025)
- [Scaling Vision Transformers for Functional MRI with Flat Maps](https://mlsanity.com/paper/2510.13768/) (2025)
- [BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics](https://mlsanity.com/paper/2606.17742/) (2026)
- [DCA: Graph-Guided Deep Embedding Clustering for Brain Atlases](https://mlsanity.com/paper/2509.01426/) (2025)
- [Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI](https://mlsanity.com/paper/2312.00236/) (2023)
- [4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis](https://mlsanity.com/paper/2504.16798/) (2025)
- [Individual brain parcellation: Review of methods, validations and applications](https://mlsanity.com/paper/2407.00984/) (2024)
- [Accelerate Coastal Ocean Circulation Model with AI Surrogate](https://mlsanity.com/paper/2410.14952/) (2024)
- [Truth Neurons](https://mlsanity.com/paper/2505.12182/) (2025)
- [Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding](https://mlsanity.com/paper/2507.22378/) (2025)
- [Progress Towards Decoding Visual Imagery via fNIRS](https://mlsanity.com/paper/2406.07662/) (2024)
- [QuantFormer: Learning to Quantize for Neural Activity Forecasting in Mouse Visual Cortex](https://mlsanity.com/paper/2412.07264/) (2024)
- [BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data](https://mlsanity.com/paper/2506.22591/) (2025)
- [Rest2Visual: Predicting Visually Evoked fMRI from Resting-State Scans](https://mlsanity.com/paper/2509.13612/) (2025)
- [Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?](https://mlsanity.com/paper/2604.03619/) (2026)
- [Swin fMRI Transformer Predicts Early Neurodevelopmental Outcomes from Neonatal fMRI](https://mlsanity.com/paper/2412.07783/) (2024)
- [Predicting Cognition from fMRI:A Comparative Study of Graph, Transformer, and Kernel Models Across Task and Rest Conditions](https://mlsanity.com/paper/2507.21016/) (2025)
- [Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI](https://mlsanity.com/paper/2605.01240/) (2026)
- [Nonlocal operator learning for fMRI encoding and decoding tasks](https://mlsanity.com/paper/2605.20389/) (2026)
- [Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI](https://mlsanity.com/paper/2606.26894/) (2026)

## Abstract (arXiv)

Modeling spatiotemporal brain dynamics from high-dimensional data, such as functional Magnetic Resonance Imaging (fMRI), is a formidable task in neuroscience. Existing approaches for fMRI analysis utilize hand-crafted features, but the process of feature extraction risks losing essential information in fMRI scans. To address this challenge, we present SwiFT (Swin 4D fMRI Transformer), a Swin Transformer architecture that can learn brain dynamics directly from fMRI volumes in a memory and computation-efficient manner. SwiFT achieves this by implementing a 4D window multi-head self-attention mechanism and absolute positional embeddings. We evaluate SwiFT using multiple large-scale resting-state fMRI datasets, including the Human Connectome Project (HCP), Adolescent Brain Cognitive Development (ABCD), and UK Biobank (UKB) datasets, to predict sex, age, and cognitive intelligence. Our experimental outcomes reveal that SwiFT consistently outperforms recent state-of-the-art models. Furthermore, by leveraging its end-to-end learning capability, we show that contrastive loss-based self-supervised pre-training of SwiFT can enhance performance on downstream tasks. Additionally, we employ an explainable AI method to identify the brain regions associated with sex classification. To our knowledge, SwiFT is the first Swin Transformer architecture to process dimensional spatiotemporal brain functional data in an end-to-end fashion. Our work holds substantial potential in facilitating scalable learning of functional brain imaging in neuroscience research by reducing the hurdles associated with applying Transformer models to high-dimensional fMRI.