# ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling

arXiv: https://arxiv.org/abs/2602.15537 · HTML: https://mlsanity.com/paper/2602.15537/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Textually Pretrained Speech Language Models](https://mlsanity.com/paper/2305.13009/) (2023)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Scaling Properties of Speech Language Models](https://mlsanity.com/paper/2404.00685/) (2024)
- [SyllableLM: Learning Coarse Semantic Units for Speech Language Models](https://mlsanity.com/paper/2410.04029/) (2024)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [Sylber: Syllabic Embedding Representation of Speech from Raw Audio](https://mlsanity.com/paper/2410.07168/) (2024)
- [SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT](https://mlsanity.com/paper/2310.10803/) (2023)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Spoken Language Modeling with Duration-Penalized Self-Supervised Units](https://mlsanity.com/paper/2505.23494/) (2025)

## Built on by (most cited first, 5 of 5)

- [Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces](https://mlsanity.com/paper/2603.12642/) (2026)
- [Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features](https://mlsanity.com/paper/2603.03096/) (2026)
- [Speech Playground: An Interactive Tool for Speech Analysis and Comparison](https://mlsanity.com/paper/2607.00418/) (2026)
- [Recovering the Zipfian Distribution in Unsupervised Term Discovery](https://mlsanity.com/paper/2606.10781/) (2026)
- [Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization](https://mlsanity.com/paper/2607.04064/) (2026)

## Abstract (arXiv)

Pure speech language models aim to learn language directly from raw audio without textual resources. A key challenge is that discrete tokens from self-supervised speech encoders result in excessively long sequences, motivating recent work on syllable-like units. However, methods like Sylber and SyllableLM rely on intricate multi-stage training pipelines. We propose ZeroSyl, a simple training-free method to extract syllable boundaries and embeddings directly from a frozen WavLM model. Using L2 norms of features in WavLM's intermediate layers, ZeroSyl achieves competitive syllable segmentation performance. The resulting segments are mean-pooled, discretized using K-means, and used to train a language model. ZeroSyl outperforms prior syllabic tokenizers across lexical, syntactic, and narrative benchmarks. Scaling experiments show that while finer-grained units are beneficial for lexical tasks, our discovered syllabic units exhibit better scaling behavior for syntactic modeling.