# Uncovering cognitive taskonomy through transfer learning in masked autoencoder-based fMRI reconstruction

arXiv: https://arxiv.org/abs/2407.00033 · HTML: https://mlsanity.com/paper/2407.00033/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [EEGdenoiseNet: A benchmark dataset for end-to-end deep learning solutions of EEG denoising](https://mlsanity.com/paper/2009.11662/) (2020)
- [Continual task learning in natural and artificial agents](https://mlsanity.com/paper/2210.04520/) (2022)
- [Explainable fMRI-based Brain Decoding via Spatial Temporal-pyramid Graph Convolutional Network](https://mlsanity.com/paper/2210.05713/) (2022)
- [HyperNTF: A Hypergraph Regularized Nonnegative Tensor Factorization for Dimensionality Reduction](https://mlsanity.com/paper/2101.06827/) (2021)
- [Transfer learning to decode brain states reflecting the relationship between cognitive tasks](https://mlsanity.com/paper/2206.03950/) (2022)
- [A Hybrid Brain-Computer Interface Using Motor Imagery and SSVEP Based on Convolutional Neural Network](https://mlsanity.com/paper/2212.05289/) (2022)

## Built on by (most cited first, 7 of 7)

- [Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining](https://mlsanity.com/paper/2604.12683/) (2026)
- [FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI](https://mlsanity.com/paper/2606.11500/) (2026)
- [BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics](https://mlsanity.com/paper/2606.17742/) (2026)
- [Beyond Single-Source Cognitive Taskonomy:Multi-Source Task Relations through fMRI Transfer Learning](https://mlsanity.com/paper/2606.26279/) (2026)
- [BrainTaskonomy: Learning How to Pretrain and What to Transfer in fMRI Foundation Models](https://mlsanity.com/paper/2609.10518/) (2026)
- [Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping](https://mlsanity.com/paper/2511.11437/) (2025)
- [BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction](https://mlsanity.com/paper/2602.04512/) (2026)

## Abstract (arXiv)

Data reconstruction is a widely used pre-training task to learn the generalized features for many downstream tasks. Although reconstruction tasks have been applied to neural signal completion and denoising, neural signal reconstruction is less studied. Here, we employ the masked autoencoder (MAE) model to reconstruct functional magnetic resonance imaging (fMRI) data, and utilize a transfer learning framework to obtain the cognitive taskonomy, a matrix to quantify the similarity between cognitive tasks. Our experimental results demonstrate that the MAE model effectively captures the temporal dynamics patterns and interactions within the brain regions, enabling robust cross-subject fMRI signal reconstruction. The cognitive taskonomy derived from the transfer learning framework reveals the relationships among cognitive tasks, highlighting subtask correlations within motor tasks and similarities between emotion, social, and gambling tasks. Our study suggests that the fMRI reconstruction with MAE model can uncover the latent representation and the obtained taskonomy offers guidance for selecting source tasks in neural decoding tasks for improving the decoding performance on target tasks.