# Speaker-Invariant Training via Adversarial Learning

arXiv: https://arxiv.org/abs/1804.00732 · HTML: https://mlsanity.com/paper/1804.00732/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 23 of 23)

- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [Speaker-adaptive Lip Reading with User-dependent Padding](https://mlsanity.com/paper/2208.04498/) (2022)
- [Modular Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2210.17049/) (2022)
- [Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition](https://mlsanity.com/paper/2302.08102/) (2023)
- [Disentangling Voice and Content with Self-Supervision for Speaker Recognition](https://mlsanity.com/paper/2310.01128/) (2023)
- [PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations](https://mlsanity.com/paper/2203.16965/) (2022)
- [Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek](https://mlsanity.com/paper/2301.00304/) (2022)
- [Robust Stuttering Detection via Multi-task and Adversarial Learning](https://mlsanity.com/paper/2204.01735/) (2022)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)
- [Enhancing and Adversarial: Improve ASR with Speaker Labels](https://mlsanity.com/paper/2211.06369/) (2022)
- [Pre-Finetuning for Few-Shot Emotional Speech Recognition](https://mlsanity.com/paper/2302.12921/) (2023)
- [Adversarial Representation Learning for Robust Privacy Preservation in Audio](https://mlsanity.com/paper/2305.00011/) (2023)
- [Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation](https://mlsanity.com/paper/2407.06310/) (2024)
- [Senone-aware Adversarial Multi-task Training for Unsupervised Child to Adult Speech Adaptation](https://mlsanity.com/paper/2102.11488/) (2021)
- [Adversarial training of Keyword Spotting to Minimize TTS Data Overfitting](https://mlsanity.com/paper/2408.10463/) (2024)
- [SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages](https://mlsanity.com/paper/2601.09050/) (2026)
- [Human-Imitating Metrics for Training and Evaluating Privacy Preserving Emotion Recognition Models Using Sociolinguistic Knowledge](https://mlsanity.com/paper/2104.08792/) (2021)
- [Speaker- and Age-Invariant Training for Child Acoustic Modeling Using Adversarial Multi-Task Learning](https://mlsanity.com/paper/2210.10231/) (2022)
- [Language-Universal Speech Attributes Modeling for Zero-Shot Multilingual Spoken Keyword Recognition](https://mlsanity.com/paper/2406.02488/) (2024)
- [SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training](https://mlsanity.com/paper/2609.02941/) (2026)
- [When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models](https://mlsanity.com/paper/2609.27382/) (2026)

## Abstract (arXiv)

We propose a novel adversarial multi-task learning scheme, aiming at actively curtailing the inter-talker feature variability while maximizing its senone discriminability so as to enhance the performance of a deep neural network (DNN) based ASR system. We call the scheme speaker-invariant training (SIT). In SIT, a DNN acoustic model and a speaker classifier network are jointly optimized to minimize the senone (tied triphone state) classification loss, and simultaneously mini-maximize the speaker classification loss. A speaker-invariant and senone-discriminative deep feature is learned through this adversarial multi-task learning. With SIT, a canonical DNN acoustic model with significantly reduced variance in its output probabilities is learned with no explicit speaker-independent (SI) transformations or speaker-specific representations used in training or testing. Evaluated on the CHiME-3 dataset, the SIT achieves 4.99% relative word error rate (WER) improvement over the conventional SI acoustic model. With additional unsupervised speaker adaptation, the speaker-adapted (SA) SIT model achieves 4.86% relative WER gain over the SA SI acoustic model.