# SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals

arXiv: https://arxiv.org/abs/2405.17766 · HTML: https://mlsanity.com/paper/2405.17766/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG](https://mlsanity.com/paper/1703.04046/) (2017)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [SeqSleepNet: End-to-End Hierarchical Recurrent Neural Network for Sequence-to-Sequence Automatic Sleep Staging](https://mlsanity.com/paper/1809.10932/) (2018)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [SleepEEGNet: Automated Sleep Stage Scoring with Sequence to Sequence Deep Learning Approach](https://mlsanity.com/paper/1903.02108/) (2019)
- [Neural network an1alysis of sleep stages enables efficient diagnosis of narcolepsy](https://mlsanity.com/paper/1710.02094/) (2017)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Automatic Sleep Stage Scoring with Single-Channel EEG Using Convolutional Neural Networks](https://mlsanity.com/paper/1610.01683/) (2016)
- [CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Electroencephalography Signal Conversion](https://mlsanity.com/paper/2311.07788/) (2023)
- [MSED: a multi-modal sleep event detection model for clinical sleep analysis](https://mlsanity.com/paper/2101.02530/) (2021)
- [Electrocardiographic Deep Learning for Predicting Post-Procedural Mortality](https://mlsanity.com/paper/2205.03242/) (2022)

## Built on by (most cited first, 30 of 41)

- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [SensorLM: Learning the Language of Wearable Sensors](https://mlsanity.com/paper/2506.09108/) (2025)
- [MIRA: Medical Time Series Foundation Model for Real-World Health Data](https://mlsanity.com/paper/2506.07584/) (2025)
- [HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series](https://mlsanity.com/paper/2510.25785/) (2025)
- [Wearable Accelerometer Foundation Models for Health via Knowledge Distillation](https://mlsanity.com/paper/2412.11276/) (2024)
- [SleepLM: Natural-Language Intelligence for Human Sleep](https://mlsanity.com/paper/2602.23605/) (2026)
- [OSF: On Pre-training and Scaling of Sleep Foundation Models](https://mlsanity.com/paper/2603.00190/) (2026)
- [Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models](https://mlsanity.com/paper/2409.11302/) (2024)
- [NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces](https://mlsanity.com/paper/2605.14698/) (2026)
- [Towards a General Intelligence and Interface for Wearable Health Data](https://mlsanity.com/paper/2605.22759/) (2026)
- [wav2sleep: A Unified Multi-Modal Approach to Sleep Stage Classification from Physiological Signals](https://mlsanity.com/paper/2411.04644/) (2024)
- [Physiology-Aware Masked Cross-Modal Reconstruction for Biosignal Representation Learning](https://mlsanity.com/paper/2605.00973/) (2026)
- [PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals](https://mlsanity.com/paper/2411.00718/) (2024)
- [Toward Foundational Model for Sleep Analysis Using a Multimodal Hybrid Self-Supervised Learning Framework](https://mlsanity.com/paper/2502.17481/) (2025)
- [OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens](https://mlsanity.com/paper/2604.18827/) (2026)
- [PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset](https://mlsanity.com/paper/2507.03165/) (2025)
- [Self-DANA: A Resource-Efficient Channel-Adaptive Self-Supervised Approach for ECG Foundation Models](https://mlsanity.com/paper/2507.14151/) (2025)
- [Stanford Sleep Bench: Evaluating Polysomnography Pre-training Methods for Sleep Foundation Models](https://mlsanity.com/paper/2512.09591/) (2025)
- [Wearable Foundation Models Should Go Beyond Static Encoders](https://mlsanity.com/paper/2603.19564/) (2026)
- [Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals](https://mlsanity.com/paper/2510.14254/) (2025)
- [SleepMaMi: A Universal Sleep Foundation Model for Integrating Macro- and Micro-structures](https://mlsanity.com/paper/2602.07628/) (2026)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Pretraining on Sleep Data Improves non-Sleep Biosignal Tasks](https://mlsanity.com/paper/2605.02500/) (2026)
- [Generating Realistic Multi-Beat ECG Signals](https://mlsanity.com/paper/2505.18189/) (2025)
- [BioX-Bridge: Model Bridging for Unsupervised Cross-Modal Knowledge Transfer across Biosignals](https://mlsanity.com/paper/2510.02276/) (2025)
- [Learning Longitudinal Health Representations from EHR and Wearable Data](https://mlsanity.com/paper/2601.12227/) (2026)
- [BiTimeCrossNet: Time-Aware Self-Supervised Learning for Pediatric Sleep](https://mlsanity.com/paper/2602.02769/) (2026)
- [Uncertainty-Aware Foundation Models for Clinical Data](https://mlsanity.com/paper/2604.04175/) (2026)
- [Multimodal Cardiovascular Risk Profiling Using Self-Supervised Learning of Polysomnography](https://mlsanity.com/paper/2507.09009/) (2025)

## Abstract (arXiv)

Sleep is a complex physiological process evaluated through various modalities recording electrical brain, cardiac, and respiratory activities. We curate a large polysomnography dataset from over 14,000 participants comprising over 100,000 hours of multi-modal sleep recordings. Leveraging this extensive dataset, we developed SleepFM, the first multi-modal foundation model for sleep analysis. We show that a novel leave-one-out approach for contrastive learning significantly improves downstream task performance compared to representations from standard pairwise contrastive learning. A logistic regression model trained on SleepFM's learned embeddings outperforms an end-to-end trained convolutional neural network (CNN) on sleep stage classification (macro AUROC 0.88 vs 0.72 and macro AUPRC 0.72 vs 0.48) and sleep disordered breathing detection (AUROC 0.85 vs 0.69 and AUPRC 0.77 vs 0.61). Notably, the learned embeddings achieve 48% top-1 average accuracy in retrieving the corresponding recording clips of other modalities from 90,000 candidates. This work demonstrates the value of holistic multi-modal sleep modeling to fully capture the richness of sleep recordings. SleepFM is open source and available at https://github.com/rthapa84/sleepfm-codebase.