# OSF: On Pre-training and Scaling of Sleep Foundation Models

arXiv: https://arxiv.org/abs/2603.00190 · HTML: https://mlsanity.com/paper/2603.00190/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Large-scale Training of Foundation Models for Wearable Biosignals](https://mlsanity.com/paper/2312.05409/) (2023)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data](https://mlsanity.com/paper/2206.02909/) (2022)
- [This Time is Different: An Observability Perspective on Time Series Foundation Models](https://mlsanity.com/paper/2505.14766/) (2025)
- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [SensorLM: Learning the Language of Wearable Sensors](https://mlsanity.com/paper/2506.09108/) (2025)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals](https://mlsanity.com/paper/2405.17766/) (2024)
- [Multi-View Spatial-Temporal Graph Convolutional Networks with Domain Generalization for Sleep Stage Classification](https://mlsanity.com/paper/2109.01824/) (2021)
- [LSM-2: Learning from Incomplete Wearable Sensor Data](https://mlsanity.com/paper/2506.05321/) (2025)
- [A Large Collection of Real-world Pediatric Sleep Studies](https://mlsanity.com/paper/2102.13284/) (2021)
- [HEARTS: Benchmarking LLM Reasoning on Health Time Series](https://mlsanity.com/paper/2603.06638/) (2026)
- [wav2sleep: A Unified Multi-Modal Approach to Sleep Stage Classification from Physiological Signals](https://mlsanity.com/paper/2411.04644/) (2024)
- [PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals](https://mlsanity.com/paper/2411.00718/) (2024)
- [Stanford Sleep Bench: Evaluating Polysomnography Pre-training Methods for Sleep Foundation Models](https://mlsanity.com/paper/2512.09591/) (2025)

## Built on by (most cited first, 12 of 12)

- [GlucoFM: A Dual-Stream Foundation Model for Continuous Glucose Monitoring](https://mlsanity.com/paper/2605.30865/) (2026)
- [Inertia-1: An Open Exploration of Wearable Motion Foundation Models](https://mlsanity.com/paper/2607.06617/) (2026)
- [TimeSRL: Generalizable Time-Series Behavioral Modeling via Semantic RL-Tuned LLMs -- A Case Study in Mental Health](https://mlsanity.com/paper/2605.21295/) (2026)
- [Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series](https://mlsanity.com/paper/2607.04245/) (2026)
- [Next-Token Prediction Learns Generalisable Representations of Sleep Physiology](https://mlsanity.com/paper/2606.09605/) (2026)
- [Learning transferable human physiology from two million hours of sleep with SleepFM-2](https://mlsanity.com/paper/2609.06849/) (2026)
- [PhysioBench: A Unified Benchmark for Physiological Signal Question Answering](https://mlsanity.com/paper/2609.20836/) (2026)
- [Comparative Analysis of State-of-the-Art Foundation Models for Sleep Analysis Under Channel Reduction](https://mlsanity.com/paper/2609.22105/) (2026)
- [SemPSG: A Semantic Channel-Aware Foundation Model for Polysomnography Analysis](https://mlsanity.com/paper/2609.36619/) (2026)
- [Sensor-Language-Action Models](https://mlsanity.com/paper/2610.08244/) (2026)
- [GlucoPRISM: Protocol-Supervised Pretraining for Cross-Cohort Glucose Transfer](https://mlsanity.com/paper/2610.12528/) (2026)
- [On Sleep-Wake Disorder Identification from Polysomnographic Representations](https://mlsanity.com/paper/2610.17451/) (2026)

## Abstract (arXiv)

Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts. There have been growing efforts to build general-purpose foundation models (FMs) for sleep physiology, but lack an in-depth understanding of the pre-training process and scaling patterns that lead to more generalizable sleep FMs. To fill this gap, we curate a massive corpus of 166,500 hours of sleep recordings from nine public sources and establish SleepBench, a comprehensive, fully open-source benchmark. Leveraging SleepBench, we systematically evaluate four families of self-supervised pre-training objectives and uncover three critical findings: (1) existing FMs fail to generalize to missing channels at inference; (2) channel-invariant feature learning is essential for pre-training; and (3) scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance.With an enhanced pre-training and scaling recipe, we introduce OSF, a family of sleep FMs that achieves state-of-the-art performance across nine datasets on diverse sleep and disease prediction tasks. Further analysis of OSF also reveals intriguing properties in sample efficiency, hierarchical aggregation, and cross-dataset scaling.