# Knowledge-guided EEG Representation Learning

arXiv: https://arxiv.org/abs/2403.03222 · HTML: https://mlsanity.com/paper/2403.03222/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [MAEEG: Masked Auto-encoder for EEG Representation Learning](https://mlsanity.com/paper/2211.02625/) (2022)
- [Self-supervised Contrastive Learning for EEG-based Sleep Staging](https://mlsanity.com/paper/2109.07839/) (2021)
- [Scaling Representation Learning from Ubiquitous ECG with State-Space Models](https://mlsanity.com/paper/2309.15292/) (2023)
- [Motor Imagery Decoding Using Ensemble Curriculum Learning and Collaborative Training](https://mlsanity.com/paper/2211.11460/) (2022)
- [A Generative Self-Supervised Framework using Functional Connectivity in fMRI Data](https://mlsanity.com/paper/2312.01994/) (2023)

## Built on by (most cited first, 7 of 7)

- [Neural Codecs as Biosignal Tokenizers](https://mlsanity.com/paper/2510.09095/) (2025)
- [Toward Fully-End-to-End Listened Speech Decoding from EEG Signals](https://mlsanity.com/paper/2406.08644/) (2024)
- [Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction](https://mlsanity.com/paper/2501.04844/) (2025)
- [Bridging Brain with Foundation Models through Self-Supervised Learning](https://mlsanity.com/paper/2506.16009/) (2025)
- [Aperiodic and Low-Frequency Spectral Bias in Reconstruction based EEG Foundation Models](https://mlsanity.com/paper/2605.26434/) (2026)
- [A Multi-dimensional Framework for Evaluating Generalization in EEG Foundation Models](https://mlsanity.com/paper/2605.28563/) (2026)
- [Best Practices in EEG Analysis: Preprocessing, Modeling, and Machine Learning](https://mlsanity.com/paper/2609.36609/) (2026)

## Abstract (arXiv)

Self-supervised learning has produced impressive results in multimedia domains of audio, vision and speech. This paradigm is equally, if not more, relevant for the domain of biosignals, owing to the scarcity of labelled data in such scenarios. The ability to leverage large-scale unlabelled data to learn robust representations could help improve the performance of numerous inference tasks on biosignals. Given the inherent domain differences between multimedia modalities and biosignals, the established objectives for self-supervised learning may not translate well to this domain. Hence, there is an unmet need to adapt these methods to biosignal analysis. In this work we propose a self-supervised model for EEG, which provides robust performance and remarkable parameter efficiency by using state space-based deep learning architecture. We also propose a novel knowledge-guided pre-training objective that accounts for the idiosyncrasies of the EEG signal. The results indicate improved embedding representation learning and downstream performance compared to prior works on exemplary tasks. Also, the proposed objective significantly reduces the amount of pre-training data required to obtain performance equivalent to prior works.