# S-JEPA: towards seamless cross-dataset transfer through dynamic spatial attention

arXiv: https://arxiv.org/abs/2403.11772 · HTML: https://mlsanity.com/paper/2403.11772/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022)
- [MAEEG: Masked Auto-encoder for EEG Representation Learning](https://mlsanity.com/paper/2211.02625/) (2022)
- [EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputs](https://mlsanity.com/paper/2402.17772/) (2024)

## Built on by (most cited first, 16 of 16)

- [REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects](https://mlsanity.com/paper/2510.21585/) (2025)
- [Denoising with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.03755/) (2024)
- [Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals](https://mlsanity.com/paper/2405.11459/) (2024)
- [T-JEPA: A Joint-Embedding Predictive Architecture for Trajectory Similarity Computation](https://mlsanity.com/paper/2406.12913/) (2024)
- [Recurrent Video Masked Autoencoders](https://mlsanity.com/paper/2512.13684/) (2025)
- [LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecasting](https://mlsanity.com/paper/2405.10093/) (2024)
- [From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Saptiotemporal Dynamics in Brain Signal Analysis](https://mlsanity.com/paper/2507.03633/) (2025)
- [Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction](https://mlsanity.com/paper/2603.16281/) (2026)
- [Review of Deep Representation Learning Techniques for Brain-Computer Interfaces and Recommendations](https://mlsanity.com/paper/2405.19345/) (2024)
- [Delta2Gamma: Band-Wise Adaptive Contrastive Learning of EEG for Alzheimer's Disease Detection](https://mlsanity.com/paper/2608.17231/) (2026)
- [ReMAP: Self-supervised learning to unveil brain representations and vulnerability](https://mlsanity.com/paper/2608.22042/) (2026)
- [Neural State Prediction: Obstructing Shortcut Learning in EEG Foundation Models](https://mlsanity.com/paper/2609.31167/) (2026)
- [What masking geometry works best for EEG foundation models?](https://mlsanity.com/paper/2609.33487/) (2026)
- [SPERA: Spherical Prior EEG Foundation Model with Geometry- and Frequency-Aware Latent Prediction](https://mlsanity.com/paper/2610.10571/) (2026)
- [Brain2Face: A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars](https://mlsanity.com/paper/2610.17356/) (2026)
- [W-JEPA: Wavelet Joint-Embedding Predictive Architecture for Wearable Activity Recognition](https://mlsanity.com/paper/2610.19411/) (2026)

## Abstract (arXiv)

Motivated by the challenge of seamless cross-dataset transfer in EEG signal processing, this article presents an exploratory study on the use of Joint Embedding Predictive Architectures (JEPAs). In recent years, self-supervised learning has emerged as a promising approach for transfer learning in various domains. However, its application to EEG signals remains largely unexplored. In this article, we introduce Signal-JEPA for representing EEG recordings which includes a novel domain-specific spatial block masking strategy and three novel architectures for downstream classification. The study is conducted on a 54 subjects dataset and the downstream performance of the models is evaluated on three different BCI paradigms: motor imagery, ERP and SSVEP. Our study provides preliminary evidence for the potential of JEPAs in EEG signal encoding. Notably, our results highlight the importance of spatial filtering for accurate downstream classification and reveal an influence of the length of the pre-training examples but not of the mask size on the downstream performance.