# FoME: A Foundation Model for EEG using Adaptive Temporal-Lateral Attention Scaling

arXiv: https://arxiv.org/abs/2409.12454 · HTML: https://mlsanity.com/paper/2409.12454/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Uncovering the structure of clinical EEG signals with self-supervised learning](https://mlsanity.com/paper/2007.16104/) (2020)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [BrainBERT: Self-supervised representation learning for intracranial recordings](https://mlsanity.com/paper/2302.14367/) (2023)
- [Self-supervised EEG Representation Learning for Automatic Sleep Staging](https://mlsanity.com/paper/2110.15278/) (2021)
- [BrainNet: Epileptic Wave Detection from SEEG with Hierarchical Graph Diffusion Learning](https://mlsanity.com/paper/2306.13101/) (2023)
- [EEG Decoding for Datasets with Heterogenous Electrode Configurations using Transfer Learning Graph Neural Networks](https://mlsanity.com/paper/2306.13109/) (2023)

## Built on by (most cited first, 16 of 16)

- [EEG Foundation Models: A Critical Review of Current Progress and Future Directions](https://mlsanity.com/paper/2507.11783/) (2025)
- [Brain Foundation Models: A Survey on Advancements in Neural Signal Processing and Brain Discovery](https://mlsanity.com/paper/2503.00580/) (2025)
- [A Simple Review of EEG Foundation Models: Datasets, Advancements and Future Perspectives](https://mlsanity.com/paper/2504.20069/) (2025)
- [NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces](https://mlsanity.com/paper/2605.14698/) (2026)
- [BioSerenity-E1: a self-supervised EEG model for medical applications](https://mlsanity.com/paper/2503.10362/) (2025)
- [ManifoldFormer: Geometric Deep Learning for Neural Dynamics on Riemannian Manifolds](https://mlsanity.com/paper/2511.16828/) (2025)
- [Subject-Aware Multi-Granularity Alignment for Zero-Shot EEG-to-Image Retrieval](https://mlsanity.com/paper/2604.17782/) (2026)
- [Transformer-based EEG Decoding: A Survey](https://mlsanity.com/paper/2507.02320/) (2025)
- [ZUNA: Flexible EEG Superresolution with Position-Aware Diffusion Autoencoders](https://mlsanity.com/paper/2602.18478/) (2026)
- [DIVER-0 : A Fully Channel Equivariant EEG Foundation Model](https://mlsanity.com/paper/2507.14141/) (2025)
- [Bridging Brain with Foundation Models through Self-Supervised Learning](https://mlsanity.com/paper/2506.16009/) (2025)
- [AnySleep: a channel-agnostic deep learning system for high-resolution sleep staging in multi-center cohorts](https://mlsanity.com/paper/2512.14461/) (2025)
- [MANAS-2: Constrained Reconstruction for EEG Foundation Models](https://mlsanity.com/paper/2609.13717/) (2026)
- [EvoBrain: Continual Learning of EEG Foundation Models Across Heterogeneous BCI Tasks](https://mlsanity.com/paper/2606.01767/) (2026)
- [EEG-PRISM: Physiologically-Grounded Interpretability of Predictions by EEG Foundation Models](https://mlsanity.com/paper/2608.13676/) (2026)
- [Representational and Functional Robustness to Electrode Montages in EEG Foundation Models](https://mlsanity.com/paper/2609.36288/) (2026)

## Abstract (arXiv)

Electroencephalography (EEG) is a vital tool to measure and record brain activity in neuroscience and clinical applications, yet its potential is constrained by signal heterogeneity, low signal-to-noise ratios, and limited labeled datasets. In this paper, we propose FoME (Foundation Model for EEG), a novel approach using adaptive temporal-lateral attention scaling to address above-mentioned challenges. FoME is pre-trained on a diverse 1.7TB dataset of scalp and intracranial EEG recordings, comprising 745M parameters trained for 1,096k steps. Our model introduces two key innovations: a time-frequency fusion embedding technique and an adaptive time-lateral attention scaling (ATLAS) mechanism. These components synergistically capture complex temporal and spectral EEG dynamics, enabling FoME to adapt to varying patterns across diverse data streams and facilitate robust multi-channel modeling. Evaluations across four downstream tasks demonstrate FoME's superior performance in classification and forecasting applications, consistently achieving state-of-the-art results. To conclude, FoME establishes a new paradigm for EEG analysis, offering a versatile foundation that advances brain-computer interfaces, clinical diagnostics, and cognitive research across neuroscience and related fields. Our code will be available at https://github.com/1061413241/FoME.