# A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Models

arXiv: https://arxiv.org/abs/2601.20896 · HTML: https://mlsanity.com/paper/2601.20896/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2026.

## Built on by (most cited first, 3 of 3)

- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation](https://mlsanity.com/paper/2606.14791/) (2026)
- [Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models](https://mlsanity.com/paper/2609.25890/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has transformed speech processing, yet its reliance on massive pre-training datasets remains a bottleneck. While robustness is often attributed to scale and diversity, the role of the data distribution is less understood. We systematically examine how curated subsets of pre-training data influence Automatic Speech Recognition (ASR) performance. Surprisingly, optimizing for acoustic, speaker, or linguistic diversity yields no clear improvements over random sampling. Instead, we find that prioritizing the longest utterances achieves superior ASR results while using only half the original dataset, reducing pre-training time by 24% on a large corpora. These findings suggest that for pre-training speech SSL models, data length is a more critical factor than either data diversity or overall data quantity for performance and efficiency, offering a new perspective for data selection strategies in SSL speech processing.