# EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

arXiv: https://arxiv.org/abs/2401.10278 · HTML: https://mlsanity.com/paper/2401.10278/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Self-Supervised Graph Neural Networks for Improved Electroencephalographic Seizure Analysis](https://mlsanity.com/paper/2104.08336/) (2021)
- [BrainBERT: Self-supervised representation learning for intracranial recordings](https://mlsanity.com/paper/2302.14367/) (2023)
- [SOM-VAE: Interpretable Discrete Representation Learning on Time Series](https://mlsanity.com/paper/1806.02199/) (2018)
- [Self-supervised Contrastive Learning for EEG-based Sleep Staging](https://mlsanity.com/paper/2109.07839/) (2021)
- [Protecting the Future: Neonatal Seizure Detection with Spatial-Temporal Modeling](https://mlsanity.com/paper/2307.05382/) (2023)
- [Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning](https://mlsanity.com/paper/2309.10910/) (2023)

## Built on by (most cited first, 30 of 52)

- [Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark](https://mlsanity.com/paper/2404.16563/) (2024)
- [EEG Foundation Models: A Critical Review of Current Progress and Future Directions](https://mlsanity.com/paper/2507.11783/) (2025)
- [LUNA: Efficient and Topology-Agnostic Foundation Model for EEG Signal Analysis](https://mlsanity.com/paper/2510.22257/) (2025)
- [Neuro-3D: Towards 3D Visual Decoding from EEG Signals](https://mlsanity.com/paper/2411.12248/) (2024)
- [Brain Foundation Models: A Survey on Advancements in Neural Signal Processing and Brain Discovery](https://mlsanity.com/paper/2503.00580/) (2025)
- [The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning](https://mlsanity.com/paper/2406.04328/) (2024)
- [FEMBA: Efficient and Scalable EEG Analysis with a Bidirectional Mamba Foundation Model](https://mlsanity.com/paper/2502.06438/) (2025)
- [CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention](https://mlsanity.com/paper/2501.10885/) (2025)
- [Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals](https://mlsanity.com/paper/2405.11459/) (2024)
- [Deeploy: Enabling Energy-Efficient Deployment of Small Language Models On Heterogeneous Microcontrollers](https://mlsanity.com/paper/2408.04413/) (2024)
- [A Simple Review of EEG Foundation Models: Datasets, Advancements and Future Perspectives](https://mlsanity.com/paper/2504.20069/) (2025)
- [Brain4FMs: A Benchmark of Foundation Models for Electrical Brain Signal](https://mlsanity.com/paper/2602.11558/) (2026)
- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)
- [Is Limited Participant Diversity Impeding EEG-based Machine Learning?](https://mlsanity.com/paper/2503.13497/) (2025)
- [Automated Contrastive Learning Strategy Search for Time Series](https://mlsanity.com/paper/2403.12641/) (2024)
- [Large Language Models for EEG: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2506.06353/) (2025)
- [DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations](https://mlsanity.com/paper/2512.19097/) (2025)
- [BioSerenity-E1: a self-supervised EEG model for medical applications](https://mlsanity.com/paper/2503.10362/) (2025)
- [From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Saptiotemporal Dynamics in Brain Signal Analysis](https://mlsanity.com/paper/2507.03633/) (2025)
- [ManifoldFormer: Geometric Deep Learning for Neural Dynamics on Riemannian Manifolds](https://mlsanity.com/paper/2511.16828/) (2025)
- [LuMamba: Latent Unified Mamba for Electrode Topology-Invariant and Efficient EEG Modeling](https://mlsanity.com/paper/2603.19100/) (2026)
- [OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens](https://mlsanity.com/paper/2604.18827/) (2026)
- [A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications](https://mlsanity.com/paper/2410.08224/) (2024)
- [Deep Learning-Powered Electrical Brain Signals Analysis: Advancing Neurological Diagnostics](https://mlsanity.com/paper/2502.17213/) (2025)
- [PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting](https://mlsanity.com/paper/2508.13773/) (2025)
- [An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling](https://mlsanity.com/paper/2502.17873/) (2025)
- [From Theory to Application: Fine-Tuning Large EEG Model with Real-World Stress Data](https://mlsanity.com/paper/2505.23042/) (2025)
- [Pretraining on Sleep Data Improves non-Sleep Biosignal Tasks](https://mlsanity.com/paper/2605.02500/) (2026)
- [NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction](https://mlsanity.com/paper/2606.24087/) (2026)
- [ADformer: A Multi-Granularity Spatial-Temporal Transformer for EEG-Based Alzheimer Detection](https://mlsanity.com/paper/2409.00032/) (2024)

## Abstract (arXiv)

Self-supervised learning has emerged as a highly effective approach in the fields of natural language processing and computer vision. It is also applicable to brain signals such as electroencephalography (EEG) data, given the abundance of available unlabeled data that exist in a wide spectrum of real-world medical applications ranging from seizure detection to wave analysis. The existing works leveraging self-supervised learning on EEG modeling mainly focus on pretraining upon each individual dataset corresponding to a single downstream task, which cannot leverage the power of abundant data, and they may derive sub-optimal solutions with a lack of generalization. Moreover, these methods rely on end-to-end model learning which is not easy for humans to understand. In this paper, we present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. The pretrained model cannot only learn universal representations on EEG signals with adaptable performance on various downstream tasks but also provide interpretable outcomes of the useful patterns within the data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings. Furthermore, we demonstrate how the learned model exhibits transferable anomaly detection performance and provides valuable interpretability of the acquired patterns via self-supervised learning.