mlsanityData to 7 Oct 2026

Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

arXiv 2607.26350 ↗First posted 28 Jul 2026cs.SD cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural audio codecs (NACs) have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be used to train self-supervised learning (SSL) models. Such models, referred to as codec-based SSL models, reduce data storage and computational cost, enabling scalable SSL pre-training. However, their language sensitivity remains unclear. When the language changes, codec-based SSL models may require retraining, which undermines their efficiency. In this paper, we present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language while keeping the other fixed. Experimental results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language. These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

26 of 34 references.

References · 26

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  9. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  11. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  12. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
14 more
  1. 2025Discrete Audio Tokens: More Than a Survey!cited by 37
  2. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 25
  3. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  4. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  5. 2021Construction of a Large-scale Japanese ASR Corpus on TV Recordingscited by 8
  6. 2023Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpuscited by 7
  7. 2025EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotationscited by 7
  8. 2025PAST: Phonetic-Acoustic Speech Tokenizercited by 6
  9. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  10. 2024How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenariocited by 5
  11. 2023Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Modelscited by 3
  12. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  13. 2025On the Language and Gender Biases in PSTN, VoIP and Neural Audio Codecscited by 1
  14. 2026Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Taskscited by 1

Built on by

Not cited yet.