# Sequential Transfer Learning to Decode Heard and Imagined Timbre from fMRI Data

arXiv: https://arxiv.org/abs/2305.13226 · HTML: https://mlsanity.com/paper/2305.13226/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 63 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [BolT: Fused Window Transformers for fMRI Time Series Analysis](https://mlsanity.com/paper/2205.11578/) (2022)
- [Attend and Decode: 4D fMRI Task State Decoding Using Attention Models](https://mlsanity.com/paper/2004.05234/) (2020)
- [Self-Supervised Transformers for fMRI representation](https://mlsanity.com/paper/2112.05761/) (2021)
- [Decoding and mapping task states of the human brain via deep learning](https://mlsanity.com/paper/1801.09858/) (2018)
- [Analyzing Neuroimaging Data Through Recurrent Deep Learning Models](https://mlsanity.com/paper/1810.09945/) (2018)
- [Learning Neural Markers of Schizophrenia Disorder Using Recurrent Neural Networks](https://mlsanity.com/paper/1712.00512/) (2017)
- [Decoding Imagined Auditory Pitch Phenomena with an Autoencoder Based Temporal Convolutional Architecture](https://mlsanity.com/paper/2305.08987/) (2023)

## Abstract (arXiv)

We present a sequential transfer learning framework for transformers on functional Magnetic Resonance Imaging (fMRI) data and demonstrate its significant benefits for decoding musical timbre. In the first of two phases, we pre-train our stacked-encoder transformer architecture on Next Thought Prediction, a self-supervised task of predicting whether or not one sequence of fMRI data follows another. This phase imparts a general understanding of the temporal and spatial dynamics of neural activity, and can be applied to any fMRI dataset. In the second phase, we fine-tune the pre-trained models and train additional fresh models on the supervised task of predicting whether or not two sequences of fMRI data were recorded while listening to the same musical timbre. The fine-tuned models achieve significantly higher accuracy with shorter training times than the fresh models, demonstrating the efficacy of our framework for facilitating transfer learning on fMRI data. Additionally, our fine-tuning task achieves a level of classification granularity beyond standard methods. This work contributes to the growing literature on transformer architectures for sequential transfer learning on fMRI data, and provides evidence that our framework is an improvement over current methods for decoding timbre.