mlsanityData to 7 Oct 2026

Improving generalizability of distilled self-supervised speech processing models under distorted settings

arXiv 2210.07978 ↗First posted 14 Oct 2022cs.SD cs.CL eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Self-supervised learned (SSL) speech pre-trained models perform well across various speech processing tasks. Distilled versions of SSL models have been developed to match the needs of on-device speech applications. Though having similar performance as original SSL models, distilled counterparts suffer from performance degradation even more than their original versions in distorted environments. This paper proposes to apply Cross-Distortion Mapping and Domain Adversarial Training to SSL models during knowledge distillation to alleviate the performance gap caused by the domain mismatch problem. Results show consistent performance improvements under both in- and out-of-domain distorted setups for different downstream tasks while keeping efficient model size.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 28 of its 32 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

28 of 32 references.

References · 28

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  11. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  12. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
16 more
  1. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  2. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  3. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  4. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  5. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  6. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  7. 2020Augmentation adversarial training for self-supervised speaker recognitioncited by 24
  8. 2022A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognitioncited by 20
  9. 2020REDAT: Accent-Invariant Representation for End-to-End ASR by Domain Adversarial Training with Relabelingcited by 18
  10. 2021Best of Both Worlds: Robust Accented Speech Recognition with Adversarial Transfer Learningcited by 17
  11. 2022Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptationcited by 16
  12. 2018Noise Adaptive Speech Enhancement using Domain Adversarial Trainingcited by 15
  13. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  14. 2021Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstructioncited by 12
  15. 2022Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASRcited by 9
  16. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 1.

5 papers.

  1. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  2. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  3. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12
  4. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 10
  5. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10