# A foundation model with multi-variate parallel attention to generate neuronal activity

arXiv: https://arxiv.org/abs/2506.20354 · HTML: https://mlsanity.com/paper/2506.20354/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2025.

## Built on by (most cited first, 6 of 6)

- [Brain4FMs: A Benchmark of Foundation Models for Electrical Brain Signal](https://mlsanity.com/paper/2602.11558/) (2026)
- [DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations](https://mlsanity.com/paper/2512.19097/) (2025)
- [iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes](https://mlsanity.com/paper/2609.06960/) (2026)
- [Pretraining for Sample-Efficient Neural Interfaces](https://mlsanity.com/paper/2609.13507/) (2026)
- [iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark](https://mlsanity.com/paper/2609.18104/) (2026)
- [PHASE: A Physiology-Guided Hierarchical Foundation Model for Intracranial EEG](https://mlsanity.com/paper/2609.36087/) (2026)

## Abstract (arXiv)

Learning from multi-variate time-series with heterogeneous channel configurations remains a fundamental challenge for deep neural networks, particularly in clinical domains such as intracranial electroencephalography (iEEG), where channel setups vary widely across subjects. In this work, we introduce multi-variate parallel attention (MVPA), a novel self-attention mechanism that disentangles content, temporal, and spatial attention, enabling flexible, generalizable, and efficient modeling of time-series data with varying channel counts and configurations. We use MVPA to build MVPFormer, a generative foundation model for human electrophysiology, trained to predict the evolution of iEEG signals across subjects. To support this and future efforts by the community, we release the SWEC iEEG dataset, the largest publicly available iEEG dataset to date, comprising nearly 10,000 hours of recordings from heterogeneous clinical sources. MVPFormer leverages MVPA to achieve strong generalization across subjects, demonstrating expert-level performance in several iEEG tasks. MVPFormer surpasses state-of-the-art (SOTA) Transformer baselines in seizure detection across the SWEC, the MAYO, and the FNUSA datasets, while also achieving SOTA performance on four Brain TreeBank iEEG decoding tasks (volume, pitch, onset, and speech). We further validate MVPA on standard time-series forecasting and classification tasks, where it matches or exceeds the performance of existing attention-based models. Together, our contributions establish MVPA as a general-purpose attention mechanism for heterogeneous time-series and MVPFormer as the first open-source, open-weights, and open-data iEEG foundation model with SOTA clinical performance. The code is available at https://github.com/IBM/multi-variate-parallel-transformer. The SWEC iEEG dataset is available at https://huggingface.co/datasets/NeuroTec/SWEC_iEEG_Dataset.