SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages
Not on the map. Why?
Abstract (arXiv)
Tonal low-resource languages are widely spoken but remain underserved by modern speech technologies. A central challenge is learning speech representations that are robust to nuisance variation, such as speaker gender, while preserving lexical tone, which carries word meaning. We propose SITA, a lightweight adaptation recipe for pretrained wav2vec-style self-supervised speech encoders. Rather than designing a new backbone or objective, SITA combines existing objectives in a staged optimization framework to reduce tone collapse while preserving ASR capability. Stage 1 improves speaker invariance without erasing tonal contrasts by combining a cross-gender contrastive loss with a tone-repulsive loss that separates same-word, different-tone realizations. Stage 2 restores recognition-oriented linguistic information through CTC fine-tuning and knowledge distillation on upper encoder layers. We evaluate SITA primarily on Hmong, a tonal language with limited digital resources and a small speaker pool. Against multilingual, speaker-adversarial, label-aware, and semi-supervised baselines, SITA achieves the best trade-off between cross-gender lexical retrieval and tone separation, while maintaining ASR accuracy close to an ASR-adapted XLS-R teacher. Results on Mandarin show consistent gains, suggesting that SITA is a general plug-in recipe for tonal speech representation learning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
23 of 55 references.
References · 23
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
11 more
- 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
- 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
- 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
- 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 71
- 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
- 2018Speaker-Invariant Training via Adversarial Learningcited by 23
- 2024Encoding of lexical tone in self-supervised models of spoken languagecited by 22
- 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
- 2018To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognitioncited by 10
- 2023Label Aware Speech Representation Learning For Language Identificationcited by 4
- 2024ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Taskscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.