mlsanityData to 14 Oct 2026

Enhancing and Adversarial: Improve ASR with Speaker Labels

arXiv 2211.06369 ↗First posted 11 Nov 2022eess.AS cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

ASR can be improved by multi-task learning (MTL) with domain enhancing or domain adversarial training, which are two opposite objectives with the aim to increase/decrease domain variance towards domain-aware/agnostic ASR, respectively. In this work, we study how to best apply these two opposite objectives with speaker labels to improve conformer-based ASR. We also propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. We also explore their combination for further improvement, achieving the same performance as i-vectors plus adversarial training. Our best speaker-based MTL achieves 7% relative improvement on the Switchboard Hub5'00 set. We also investigate the effect of such speaker-based MTL w.r.t. cleaner dataset and weaker ASR NN.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

17 of 34 references.

References · 17

  1. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2016Domain Separation Networkscited by 320
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  5. 2017Adversarial Dropout Regularizationcited by 106
  6. 2017English Conversational Telephone Speech Recognition by Humans and Machinescited by 40
  7. 2018Domain Adversarial Training for Accented Speech Recognitioncited by 40
  8. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
  9. 2018Speaker-Invariant Training via Adversarial Learningcited by 23
  10. 2021Best of Both Worlds: Robust Accented Speech Recognition with Adversarial Transfer Learningcited by 18
  11. 2018Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognitioncited by 14
  12. 2022Efficient Training of Neural Transducer for Speech Recognitioncited by 14
5 more
  1. 2016Invariant Representations for Noisy Speech Recognitioncited by 12
  2. 2019Cumulative Adaptation for BLSTM Acoustic Modelscited by 11
  3. 2017Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognitioncited by 10
  4. 2022Improving the Training Recipe for a Robust Conformer-based Hybrid Modelcited by 9
  5. 2018Adversarial Learning of Raw Speech Features for Domain Invariant Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 3.

6 papers.

  1. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  2. 2023Accented Speech Recognition With Accent-specific Codebookscited by 6
  3. 2026Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition modelscited by 2
  4. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  5. 2026Where Do Self-Supervised Speech Models Become Unfair?cited by 1
  6. 2026Speaker Group Encoding in Self-supervised Speech Recognition Modelscited by 1