# Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'a

arXiv: https://arxiv.org/abs/2604.07467 · HTML: https://mlsanity.com/paper/2604.07467/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 23 of 23)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [Self-Supervised Speech Representations are More Phonetic than Semantic](https://mlsanity.com/paper/2406.08619/) (2024)
- [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](https://mlsanity.com/paper/2204.02967/) (2022)
- [Toward Joint Language Modeling for Speech Units and Text](https://mlsanity.com/paper/2310.08715/) (2023)
- [RepCodec: A Speech Representation Codec for Speech Tokenization](https://mlsanity.com/paper/2309.00169/) (2023)
- [Encoding of lexical tone in self-supervised models of spoken language](https://mlsanity.com/paper/2403.16865/) (2024)
- [Textless Direct Speech-to-Speech Translation with Discrete Speech Representation](https://mlsanity.com/paper/2211.00115/) (2022)
- [AfriHuBERT: A self-supervised speech representation model for African languages](https://mlsanity.com/paper/2409.20201/) (2024)
- [DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding](https://mlsanity.com/paper/2406.09345/) (2024)
- [SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?](https://mlsanity.com/paper/2306.08374/) (2023)
- [ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis](https://mlsanity.com/paper/2406.08989/) (2024)
- [Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information](https://mlsanity.com/paper/2505.15667/) (2025)
- [Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning](https://mlsanity.com/paper/2506.02584/) (2025)
- [Perception of Phonological Assimilation by Neural Speech Recognition Models](https://mlsanity.com/paper/2406.15265/) (2024)
- [Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?](https://mlsanity.com/paper/2410.19935/) (2024)
- [An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training](https://mlsanity.com/paper/2509.05359/) (2025)

## Built on by (most cited first, 1 of 1)

- [Tone on a Budget: A Reference-Free Metric for Lexical Tone in Massively Multilingual Text-to-Speech](https://mlsanity.com/paper/2609.14817/) (2026)

## Abstract (arXiv)

Discrete speech units (DSUs) are derived by quantising representations from models trained using self-supervised learning (SSL). They are a popular representation for a wide variety of spoken language tasks, including those where prosody matters. DSUs are especially convenient for tasks where text and speech are jointly modelled, such as text-to-speech and multimodal dialogue systems. But we have found that DSUs encode suprasegmental information less reliably than segmental structure, which we demonstrate in this work using lexical tone, though this limitation likely extends to other suprasegmental features such as prosody. Our investigations using the tone languages Mandarin and Yor`ub'a show that the SSL latent representations themselves do encode tone, yet DSUs obtained using quantisation tend to prioritise phonetic structure, which makes lexical tone less reliably encoded. This remains true for a variety of quantisation methods, not only the most common, K-means. We conclude that current DSU quantisation strategies have limitations for suprasegmental features, which suggests a need for new, tone-aware (or prosody-aware) techniques in speech representation learning. We point towards a potential form of the solution by performing K-means clustering once to encode phonetic information, then again on the residual representation, which better encodes lexical tone.