Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?
Not on the map. Why?
Abstract (arXiv)
Discrete representations of speech, obtained from Self-Supervised Learning (SSL) foundation models, are widely used, especially where there are limited data for the downstream task, such as for a low-resource language. Typically, discretization of speech into a sequence of symbols is achieved by unsupervised clustering of the latents from an SSL model. Our study evaluates whether discrete symbols - found using k-means - adequately capture tone in two example languages, Mandarin and Yoruba. We compare latent vectors with discrete symbols, obtained from HuBERT base, MandarinHuBERT, or XLS-R, for vowel and tone classification. We find that using discrete symbols leads to a substantial loss of tone information, even for language-specialised SSL models. We suggest that discretization needs to be task-aware, particularly for tone-dependent downstream tasks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 25 Oct 2024.
Builds on
23 of 37 references.
References · 23
- 2017Neural Discrete Representation Learningcited by 4,650
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
- 2021Generative Spoken Language Modeling from Raw Audiocited by 336
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
- 2021Direct speech-to-speech translation with discrete unitscited by 148
- 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
- 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
- 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
- 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
- 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
11 more
- 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
- 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 35
- 2022BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpuscited by 31
- 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
- 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
- 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
- 2024Encoding of lexical tone in self-supervised models of spoken languagecited by 22
- 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 19
- 2022Self-Supervised Learning for Speech Enhancement through Synthesiscited by 10
- 2023Rhythm Modeling for Voice Conversioncited by 6
- 2024ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesiscited by 6
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.