mlsanityData to 7 Oct 2026

Encoding of lexical tone in self-supervised models of spoken language

arXiv 2403.16865 ↗First posted 25 Mar 2024cs.CL eess.AScited by 22

Not on the map. Why?

Abstract (arXiv)

Interpretability research has shown that self-supervised Spoken Language Models (SLMs) encode a wide variety of features in human speech from the acoustic, phonetic, phonological, syntactic and semantic levels, to speaker characteristics. The bulk of prior research on representations of phonology has focused on segmental features such as phonemes; the encoding of suprasegmental phonology (such as tone and stress patterns) in SLMs is not yet well understood. Tone is a suprasegmental feature that is present in more than half of the world's languages. This paper aims to analyze the tone encoding capabilities of SLMs, using Mandarin and Vietnamese as case studies. We show that SLMs encode lexical tone to a significant degree even when they are trained on data from non-tonal languages. We further find that SLMs behave similarly to native and non-native human participants in tone and consonant perception studies, but they do not follow the same developmental trajectory.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 23 of its 66 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

23 of 66 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  9. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  11. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  12. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
11 more
  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  2. 2015THCHS-30 : A Free Chinese Speech Corpuscited by 43
  3. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 35
  4. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 31
  5. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  6. 2021Phone-to-audio alignment without text: A Semi-supervised Approachcited by 25
  7. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  8. 2020Neural Representations for Modeling Variation in Speechcited by 14
  9. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  10. 2021Automatic recognition of suprasegmentals in speechcited by 9
  11. 2021How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddingscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  2. 2025What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-trainingcited by 4
  3. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  4. 2025Different Speech Translation Models Encode and Translate Speaker Gender Differentlycited by 3
  5. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  6. 2026Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representationscited by 3
  7. 2025Tone recognition in low-resource languages of North-East India: peeling the layers of SSL-based speech modelscited by 1
  8. 2025LTA-L2S: Lexical Tone-Aware Lip-to-Speech Synthesis for Mandarin with Cross-Lingual Transfer Learningcited by 1
  9. 2025How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfercited by 1
  10. 2026SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languagescited by 1
  11. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'acited by 1
  12. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
10 more
  1. 2025Large language models have learned to use languagecited by 0
  2. 2025Large language models are not about natural languagecited by 0
  3. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  4. 2026Are Language Models Models?cited by 0
  5. 2026Tracking the emergence of linguistic structure in self-supervised models learning from speechcited by 0
  6. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  7. 2026You Can't Fight in Here! This is BBS!cited by 0
  8. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  9. 2026Perceptual compensation for tonal context in self-supervised speech modelscited by 0
  10. 2026T-SANDHI: Tone Sandhi-aware Adaptive Network with Decoupled Hybrid Injection for Low-resource Taiwanese Hokkien Speech Recognitioncited by 0