# SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers

arXiv: https://arxiv.org/abs/2404.03831 · HTML: https://mlsanity.com/paper/2404.03831/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 77 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation](https://mlsanity.com/paper/1809.09478/) (2018)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification](https://mlsanity.com/paper/2105.11043/) (2021)
- [EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement](https://mlsanity.com/paper/2110.04447/) (2021)
- [Fast Optical Flow using Dense Inverse Search](https://mlsanity.com/paper/1603.03590/) (2016)
- [Camera Measurement of Physiological Vital Signs](https://mlsanity.com/paper/2111.11547/) (2021)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions](https://mlsanity.com/paper/2111.08446/) (2021)
- [Non-Contrastive Unsupervised Learning of Physiological Signals from Video](https://mlsanity.com/paper/2303.07944/) (2023)
- [SleepPPG-Net: a deep learning algorithm for robust sleep staging from continuous photoplethysmography](https://mlsanity.com/paper/2202.05735/) (2022)
- [Sleep Staging from Electrocardiography and Respiration with Deep Learning](https://mlsanity.com/paper/1908.11463/) (2019)
- [Algorithmic Principles of Camera-based Respiratory Motion Extraction](https://mlsanity.com/paper/2105.07537/) (2021)
- [Deep Learning-Enabled Sleep Staging From Vital Signs and Activity Measured Using a Near-Infrared Video Camera](https://mlsanity.com/paper/2306.03711/) (2023)

## Built on by (most cited first, 4 of 4)

- [wav2sleep: A Unified Multi-Modal Approach to Sleep Stage Classification from Physiological Signals](https://mlsanity.com/paper/2411.04644/) (2024)
- [sleep2vec: Unified Cross-Modal Alignment for Heterogeneous Nocturnal Biosignals](https://mlsanity.com/paper/2602.13857/) (2026)
- [Rethinking PPG-based Sleep Staging: Datasets, Metrics, and Benchmarks](https://mlsanity.com/paper/2608.00943/) (2026)
- [Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging](https://mlsanity.com/paper/2609.05550/) (2026)

## Abstract (arXiv)

Advances in camera-based physiological monitoring have enabled the robust, non-contact measurement of respiration and the cardiac pulse, which are known to be indicative of the sleep stage. This has led to research into camera-based sleep monitoring as a promising alternative to "gold-standard" polysomnography, which is cumbersome, expensive to administer, and hence unsuitable for longer-term clinical studies. In this paper, we introduce SleepVST, a transformer model which enables state-of-the-art performance in camera-based sleep stage classification (sleep staging). After pre-training on contact sensor data, SleepVST outperforms existing methods for cardio-respiratory sleep staging on the SHHS and MESA datasets, achieving total Cohen's kappa scores of 0.75 and 0.77 respectively. We then show that SleepVST can be successfully transferred to cardio-respiratory waveforms extracted from video, enabling fully contact-free sleep staging. Using a video dataset of 50 nights, we achieve a total accuracy of 78.8% and a Cohen's kappa of 0.71 in four-class video-based sleep staging, setting a new state-of-the-art in the domain.