# SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages

arXiv: https://arxiv.org/abs/2601.09050 · HTML: https://mlsanity.com/paper/2601.09050/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data](https://mlsanity.com/paper/1709.07902/) (2017)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning](https://mlsanity.com/paper/2010.13991/) (2020)
- [Speaker-Invariant Training via Adversarial Learning](https://mlsanity.com/paper/1804.00732/) (2018)
- [Encoding of lexical tone in self-supervised models of spoken language](https://mlsanity.com/paper/2403.16865/) (2024)
- [A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models](https://mlsanity.com/paper/2408.13678/) (2024)
- [To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition](https://mlsanity.com/paper/1812.03483/) (2018)
- [Label Aware Speech Representation Learning For Language Identification](https://mlsanity.com/paper/2306.04374/) (2023)
- [ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks](https://mlsanity.com/paper/2407.21066/) (2024)

## Built on by (most cited first, 1 of 1)

- [MAST: Label-Efficient, Robust, and Generalizable Sound Detection for Biodiversity Monitoring via Masked Audio Pretraining and Self-Training](https://mlsanity.com/paper/2609.15221/) (2026)

## Abstract (arXiv)

Tonal low-resource languages are widely spoken but remain underserved by modern speech technologies. A central challenge is learning speech representations that are robust to nuisance variation, such as speaker gender, while preserving lexical tone, which carries word meaning. We propose SITA, a lightweight adaptation recipe for pretrained wav2vec-style self-supervised speech encoders. Rather than designing a new backbone or objective, SITA combines existing objectives in a staged optimization framework to reduce tone collapse while preserving ASR capability. Stage 1 improves speaker invariance without erasing tonal contrasts by combining a cross-gender contrastive loss with a tone-repulsive loss that separates same-word, different-tone realizations. Stage 2 restores recognition-oriented linguistic information through CTC fine-tuning and knowledge distillation on upper encoder layers. We evaluate SITA primarily on Hmong, a tonal language with limited digital resources and a small speaker pool. Against multilingual, speaker-adversarial, label-aware, and semi-supervised baselines, SITA achieves the best trade-off between cross-gender lexical retrieval and tone separation, while maintaining ASR accuracy close to an ASR-adapted XLS-R teacher. Results on Mandarin show consistent gains, suggesting that SITA is a general plug-in recipe for tonal speech representation learning.