# SLEEPYLAND: trust begins with fair evaluation of automatic sleep staging models

arXiv: https://arxiv.org/abs/2506.08574 · HTML: https://mlsanity.com/paper/2506.08574/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 57 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification](https://mlsanity.com/paper/2105.11043/) (2021)
- [Neural network an1alysis of sleep stages enables efficient diagnosis of narcolepsy](https://mlsanity.com/paper/1710.02094/) (2017)
- [L-SeqSleepNet: Whole-cycle Long Sequence Modelling for Automatic Sleep Staging](https://mlsanity.com/paper/2301.03441/) (2023)
- [A Large Collection of Real-world Pediatric Sleep Studies](https://mlsanity.com/paper/2102.13284/) (2021)
- [Automatic sleep stage classification with deep residual networks in a mixed-cohort setting](https://mlsanity.com/paper/2008.09416/) (2020)
- [U-Sleep's resilience to AASM guidelines](https://mlsanity.com/paper/2209.11173/) (2022)
- [Multi-Scored Sleep Databases: How to Exploit the Multiple-Labels in Automated Sleep Scoring](https://mlsanity.com/paper/2207.01910/) (2022)
- [Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review](https://mlsanity.com/paper/2312.14996/) (2023)

## Built on by (most cited first, 8 of 8)

- [NAPS: Attention-Based Fusion of Heterogeneous Physiological Signals](https://mlsanity.com/paper/2511.03488/) (2025)
- [AnySleep: a channel-agnostic deep learning system for high-resolution sleep staging in multi-center cohorts](https://mlsanity.com/paper/2512.14461/) (2025)
- [Next-Token Prediction Learns Generalisable Representations of Sleep Physiology](https://mlsanity.com/paper/2606.09605/) (2026)
- [SleepVLM: A Rule-Grounded Vision-Language Model for Auditable Sleep Staging](https://mlsanity.com/paper/2603.26738/) (2026)
- [Towards Multi-Source Domain Generalization for Sleep Staging with Noisy Labels](https://mlsanity.com/paper/2604.10009/) (2026)
- [Large language models in medical time series analysis](https://mlsanity.com/paper/2609.22262/) (2026)
- [TAM-Chain: Multi-Scale Thyroid Cytology Classification via Absorbing Markov Chains and Shannon Entropy Uncertainty Quantification for False-Negative Suppression and Domain-Shift Adaptation](https://mlsanity.com/paper/2609.28590/) (2026)
- [On Sleep-Wake Disorder Identification from Polysomnographic Representations](https://mlsanity.com/paper/2610.17451/) (2026)

## Abstract (arXiv)

Despite advances in deep learning for automatic sleep staging, clinical adoption remains limited due to challenges in fair model evaluation, generalization across diverse datasets, model bias, and variability in human annotations. We present SLEEPYLAND, an open-source sleep staging evaluation framework designed to address these barriers. It includes more than 220'000 hours in-domain (ID) sleep recordings, and more than 84'000 hours out-of-domain (OOD) sleep recordings, spanning a broad range of ages, sleep-wake disorders, and hardware setups. We release pre-trained models based on high-performing SoA architectures and evaluate them under standardized conditions across single- and multi-channel EEG/EOG configurations. We introduce SOMNUS, an ensemble combining models across architectures and channel setups via soft voting. SOMNUS achieves robust performance across twenty-four different datasets, with macro-F1 scores between 68.7% and 87.2%, outperforming individual models in 94.9% of cases. Notably, SOMNUS surpasses previous SoA methods, even including cases where compared models were trained ID while SOMNUS treated the same data as OOD. Using a subset of the BSWR (N=6'633), we quantify model biases linked to age, gender, AHI, and PLMI, showing that while ensemble improves robustness, no model architecture consistently minimizes bias in performance and clinical markers estimation. In evaluations on OOD multi-annotated datasets (DOD-H, DOD-O), SOMNUS exceeds the best human scorer, i.e., MF1 85.2% vs 80.8% on DOD-H, and 80.2% vs 75.9% on DOD-O, better reproducing the scorer consensus than any individual expert (k = 0.89/0.85 and ACS = 0.95/0.94 for healthy/OSA cohorts). Finally, we introduce ensemble disagreement metrics - entropy and inter-model divergence based - predicting regions of scorer disagreement with ROC AUCs up to 0.828, offering a data-driven proxy for human uncertainty.