# Pretraining on Sleep Data Improves non-Sleep Biosignal Tasks

arXiv: https://arxiv.org/abs/2605.02500 · HTML: https://mlsanity.com/paper/2605.02500/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 4 May 2026.

## Builds on (most cited first, 21 of 21)

- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [Automatic diagnosis of the 12-lead ECG using a deep neural network](https://mlsanity.com/paper/1904.01949/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects](https://mlsanity.com/paper/2510.21585/) (2025)
- [BrainBERT: Self-supervised representation learning for intracranial recordings](https://mlsanity.com/paper/2302.14367/) (2023)
- [EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model](https://mlsanity.com/paper/2401.10278/) (2024)
- [SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals](https://mlsanity.com/paper/2405.17766/) (2024)
- [Improved EEG Event Classification Using Differential Energy](https://mlsanity.com/paper/1801.02477/) (2018)
- [EEGMamba: Bidirectional State Space Model with Mixture of Experts for EEG Multi-task Classification](https://mlsanity.com/paper/2407.20254/) (2024)
- [FEMBA: Efficient and Scalable EEG Analysis with a Bidirectional Mamba Foundation Model](https://mlsanity.com/paper/2502.06438/) (2025)
- [OpenECG: Benchmarking ECG Foundation Models with Public 1.2 Million Records](https://mlsanity.com/paper/2503.00711/) (2025)
- [Advancing the State-of-the-Art for ECG Analysis through Structured State Space Models](https://mlsanity.com/paper/2211.07579/) (2022)
- [SPEED: Scalable Preprocessing of EEG Data for Self-Supervised Learning](https://mlsanity.com/paper/2408.08065/) (2024)
- [Toward Foundational Model for Sleep Analysis Using a Multimodal Hybrid Self-Supervised Learning Framework](https://mlsanity.com/paper/2502.17481/) (2025)
- [A Deep Generative Model for Five-Class Sleep Staging with Arbitrary Sensor Input](https://mlsanity.com/paper/2408.15253/) (2024)
- [Stanford Sleep Bench: Evaluating Polysomnography Pre-training Methods for Sleep Foundation Models](https://mlsanity.com/paper/2512.09591/) (2025)

## Built on by (most cited first, 2 of 2)

- [Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders](https://mlsanity.com/paper/2605.13930/) (2026)
- [Learning transferable human physiology from two million hours of sleep with SleepFM-2](https://mlsanity.com/paper/2609.06849/) (2026)

## Abstract (arXiv)

Sleep foundation models have recently demonstrated strong performance on in-domain polysomnography tasks, including sleep staging, apnea detection, and disease risk prediction. In this work, we investigate whether sleep biosignals can serve as an effective pretraining distribution for learning representations that transfer beyond sleep to adjacent domains. Following sleep foundation models, we perform sleep-only multimodal contrastive pretraining (with a leave-one-out objective) and evaluate transfer to non-sleep EEG and ECG, two well-benchmarked biosignal modalities with heterogeneous datasets and clinically meaningful downstream tasks. Across eight downstream tasks spanning multiple EEG and ECG datasets, sleep pretraining consistently improves performance relative to training from scratch. Moreover, on several tasks, we achieve performance competitive with or surpassing prior specialized state-of-the-art and foundation models.