# wav2sleep: A Unified Multi-Modal Approach to Sleep Stage Classification from Physiological Signals

arXiv: https://arxiv.org/abs/2411.04644 · HTML: https://mlsanity.com/paper/2411.04644/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [SeqSleepNet: End-to-End Hierarchical Recurrent Neural Network for Sequence-to-Sequence Automatic Sleep Staging](https://mlsanity.com/paper/1809.10932/) (2018)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions](https://mlsanity.com/paper/2111.08446/) (2021)
- [SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals](https://mlsanity.com/paper/2405.17766/) (2024)
- [Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words](https://mlsanity.com/paper/2309.16108/) (2023)
- [SleepPPG-Net: a deep learning algorithm for robust sleep staging from continuous photoplethysmography](https://mlsanity.com/paper/2202.05735/) (2022)
- [Toward Interpretable Sleep Stage Classification Using Cross-Modal Transformers](https://mlsanity.com/paper/2208.06991/) (2022)
- [CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities](https://mlsanity.com/paper/2304.06485/) (2023)
- [SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers](https://mlsanity.com/paper/2404.03831/) (2024)

## Built on by (most cited first, 5 of 5)

- [SleepLM: Natural-Language Intelligence for Human Sleep](https://mlsanity.com/paper/2602.23605/) (2026)
- [OSF: On Pre-training and Scaling of Sleep Foundation Models](https://mlsanity.com/paper/2603.00190/) (2026)
- [BiTimeCrossNet: Time-Aware Self-Supervised Learning for Pediatric Sleep](https://mlsanity.com/paper/2602.02769/) (2026)
- [Next-Token Prediction Learns Generalisable Representations of Sleep Physiology](https://mlsanity.com/paper/2606.09605/) (2026)
- [Comparative Analysis of State-of-the-Art Foundation Models for Sleep Analysis Under Channel Reduction](https://mlsanity.com/paper/2609.22105/) (2026)

## Abstract (arXiv)

Accurate classification of sleep stages from less obtrusive sensor measurements such as the electrocardiogram (ECG) or photoplethysmogram (PPG) could enable important applications in sleep medicine. Existing approaches to this problem have typically used deep learning models designed and trained to operate on one or more specific input signals. However, the datasets used to develop these models often do not contain the same sets of input signals. Some signals, particularly PPG, are much less prevalent than others, and this has previously been addressed with techniques such as transfer learning. Additionally, only training on one or more fixed modalities precludes cross-modal information transfer from other sources, which has proved valuable in other problem domains. To address this, we introduce wav2sleep, a unified model designed to operate on variable sets of input signals during training and inference. After jointly training on over 10,000 overnight recordings from six publicly available polysomnography datasets, including SHHS and MESA, wav2sleep outperforms existing sleep stage classification models across test-time input combinations including ECG, PPG, and respiratory signals.