# k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning

arXiv: https://arxiv.org/abs/2411.17100 · HTML: https://mlsanity.com/paper/2411.17100/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2024.

## Built on by (most cited first, 6 of 6)

- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining](https://mlsanity.com/paper/2505.21527/) (2025)
- [Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations](https://mlsanity.com/paper/2604.02102/) (2026)
- [WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition](https://mlsanity.com/paper/2604.25611/) (2026)
- [Augmenting Dysarthric Speech Severity Assessment with MOS Supervision](https://mlsanity.com/paper/2606.18645/) (2026)
- [MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2607.24030/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has achieved great success in speech-related tasks. While Transformer and Conformer architectures have dominated SSL backbones, encoders like Zipformer, which excel in automatic speech recognition (ASR), remain unexplored in SSL. Concurrently, inefficiencies in data processing within existing SSL training frameworks, such as fairseq, pose challenges in managing the growing volumes of training data. To address these issues, we propose k2SSL, an open-source framework that offers faster, more memory-efficient, and better-performing self-supervised speech representation learning, focusing on downstream ASR tasks. The optimized HuBERT and proposed Zipformer-based SSL systems exhibit substantial reductions in both training time and memory usage during SSL training. Experiments on LibriSpeech demonstrate that Zipformer Base significantly outperforms HuBERT and WavLM, achieving up to a 34.8% relative WER reduction compared to HuBERT Base after fine-tuning, along with a 3.5x pre-training speedup in GPU hours. When scaled to 60k hours of LibriLight data, Zipformer Large exhibits remarkable efficiency, matching HuBERT Large's performance while requiring only 5/8 pre-training steps.