# EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputs

arXiv: https://arxiv.org/abs/2402.17772 · HTML: https://mlsanity.com/paper/2402.17772/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 17 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [MAEEG: Masked Auto-encoder for EEG Representation Learning](https://mlsanity.com/paper/2211.02625/) (2022)
- [Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2109.09559/) (2021)
- [Improving Position Encoding of Transformers for Multivariate Time Series Classification](https://mlsanity.com/paper/2305.16642/) (2023)
- [Understanding Masked Autoencoders via Hierarchical Latent Variable Models](https://mlsanity.com/paper/2306.04898/) (2023)
- [Reverse Engineering Self-Supervised Learning](https://mlsanity.com/paper/2305.15614/) (2023)
- [Improved EEG Event Classification Using Differential Energy](https://mlsanity.com/paper/1801.02477/) (2018)
- [BrainNet: Epileptic Wave Detection from SEEG with Hierarchical Graph Diffusion Learning](https://mlsanity.com/paper/2306.13101/) (2023)
- [Self-supervised Learning for Electroencephalogram: A Systematic Survey](https://mlsanity.com/paper/2401.05446/) (2024)
- [Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification](https://mlsanity.com/paper/2312.03998/) (2023)
- [Self-supervised Group Meiosis Contrastive Learning for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2208.00877/) (2022)

## Built on by (most cited first, 30 of 50)

- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects](https://mlsanity.com/paper/2510.21585/) (2025)
- [CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2506.23075/) (2025)
- [CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Model](https://mlsanity.com/paper/2506.09110/) (2025)
- [LUNA: Efficient and Topology-Agnostic Foundation Model for EEG Signal Analysis](https://mlsanity.com/paper/2510.22257/) (2025)
- [EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models](https://mlsanity.com/paper/2508.17742/) (2025)
- [Tokenizing Single-Channel EEG with Time-Frequency Motif Learning](https://mlsanity.com/paper/2502.16060/) (2025)
- [S-JEPA: towards seamless cross-dataset transfer through dynamic spatial attention](https://mlsanity.com/paper/2403.11772/) (2024)
- [Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals](https://mlsanity.com/paper/2405.11459/) (2024)
- [LEAD: An EEG Foundation Model for Alzheimer's Disease Detection](https://mlsanity.com/paper/2502.01678/) (2025)
- [DeeperBrain: A Neuro-Grounded EEG Foundation Model Towards Universal BCI](https://mlsanity.com/paper/2601.06134/) (2026)
- [Brain4FMs: A Benchmark of Foundation Models for Electrical Brain Signal](https://mlsanity.com/paper/2602.11558/) (2026)
- [Graph Adapter of EEG Foundation Models for Parameter Efficient Fine Tuning](https://mlsanity.com/paper/2411.16155/) (2024)
- [BioSerenity-E1: a self-supervised EEG model for medical applications](https://mlsanity.com/paper/2503.10362/) (2025)
- [From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Saptiotemporal Dynamics in Brain Signal Analysis](https://mlsanity.com/paper/2507.03633/) (2025)
- [SAMBA: Toward a Long-Context EEG Foundation Model via Spatial Embedding and Differential Mamba](https://mlsanity.com/paper/2511.18571/) (2025)
- [LuMamba: Latent Unified Mamba for Electrode Topology-Invariant and Efficient EEG Modeling](https://mlsanity.com/paper/2603.19100/) (2026)
- [Medical world models: representing medical states, modelling clinical dynamics and guiding intervention policies](https://mlsanity.com/paper/2606.16721/) (2026)
- [Review of Deep Representation Learning Techniques for Brain-Computer Interfaces and Recommendations](https://mlsanity.com/paper/2405.19345/) (2024)
- [BioMamba: Leveraging Spectro-Temporal Embedding in Bidirectional Mamba for Enhanced Biosignal Classification](https://mlsanity.com/paper/2503.11741/) (2025)
- [ALFEE: Adaptive Large Foundation Model for EEG Representation](https://mlsanity.com/paper/2505.06291/) (2025)
- [What Do EEG Foundation Models Capture from Human Brain Signals?](https://mlsanity.com/paper/2605.11410/) (2026)
- [CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning](https://mlsanity.com/paper/2412.11695/) (2024)
- [An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling](https://mlsanity.com/paper/2502.17873/) (2025)
- [DIVER-0 : A Fully Channel Equivariant EEG Foundation Model](https://mlsanity.com/paper/2507.14141/) (2025)
- [EEGDM: Learning EEG Representation with Latent Diffusion Model](https://mlsanity.com/paper/2508.20705/) (2025)
- [SCOPE: Structured Prototype-Guided Adaptation for EEG Foundation Models with Limited Labels](https://mlsanity.com/paper/2602.17251/) (2026)
- [FEMBA on the Edge: Physiologically-Aware Pre-Training, Quantization, and Deployment of a Bidirectional Mamba EEG Foundation Model on an Ultra-low Power Microcontroller](https://mlsanity.com/paper/2603.26716/) (2026)
- [CaMBRAIN: Real-time, Continuous EEG Inference with Causal State Space Models](https://mlsanity.com/paper/2605.28792/) (2026)
- [MSBraM: A Multi-scale Self-supervised Brain Foundation Model for Hierarchical EEG Dynamics Learning](https://mlsanity.com/paper/2607.21402/) (2026)

## Abstract (arXiv)

Self-supervised approaches for electroencephalography (EEG) representation learning face three specific challenges inherent to EEG data: (1) The low signal-to-noise ratio which challenges the quality of the representation learned, (2) The wide range of amplitudes from very small to relatively large due to factors such as the inter-subject variability, risks the models to be dominated by higher amplitude ranges, and (3) The absence of explicit segmentation in the continuous-valued sequences which can result in less informative representations. To address these challenges, we introduce textitEEG2Rep, a self-prediction approach for self-supervised representation learning from EEG. Two core novel components of EEG2Rep are as follows: 1) Instead of learning to predict the masked input from raw EEG, EEG2Rep learns to predict masked input in latent representation space, and 2) Instead of conventional masking methods, EEG2Rep uses a new semantic subsequence preserving (SSP) method which provides informative masked inputs to guide EEG2Rep to generate rich semantic representations. In experiments on 6 diverse EEG tasks with subject variability, EEG2Rep significantly outperforms state-of-the-art methods. We show that our semantic subsequence preserving improves the existing masking methods in self-prediction literature and find that preserving 50% of EEG recordings will result in the most accurate results on all 6 tasks on average. Finally, we show that EEG2Rep is robust to noise addressing a significant challenge that exists in EEG data. Models and code are available at:urlhttps://github.com/Navidfoumani/EEG2Rep