mlsanityData to 14 Oct 2026

Noise robust distillation of self-supervised speech models via correlation metrics

arXiv 2312.12153 ↗First posted 19 Dec 2023cs.SD eess.AScited by 0

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Compared to large speech foundation models, small distilled models exhibit degraded noise robustness. The student's robustness can be improved by introducing noise at the inputs during pre-training. Despite this, using the standard distillation loss still yields a student with degraded performance. Thus, this paper proposes improving student robustness via distillation with correlation metrics. Teacher behavior is learned by maximizing the teacher and student cross-correlation matrix between their representations towards identity. Noise robustness is encouraged via the student's self-correlation minimization. The proposed method is agnostic of the teacher model and consistently outperforms the previous approach. This work also proposes an heuristic to weigh the importance of the two correlation terms automatically. Experiments show consistently better clean and noise generalization on Intent Classification, Keyword Spotting, and Automatic Speech Recognition tasks on SUPERB Challenge.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 28 of its 37 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

28 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Ensemble knowledge distillation of self-supervised speech models24 Feb

Other references · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  8. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  12. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
15 more
  1. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  2. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  3. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  4. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  5. 2017Large-Scale Domain Adaptation via Teacher-Student Learningcited by 31
  6. 2018Adversarial Teacher-Student Learning for Unsupervised Domain Adaptationcited by 28
  7. 2022A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognitioncited by 22
  8. 2022Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptationcited by 17
  9. 2021Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstructioncited by 13
  10. 2022Improving generalizability of distilled self-supervised speech processing models under distorted settingscited by 7
  11. 2023deHuBERT: Disentangling Noise in a Self-supervised Model for Robust Speech Recognitioncited by 7
  12. 2022Redundancy Reduction Twins Network: A Training framework for Multi-output Emotion Regressioncited by 5
  13. 2022Audio Barlow Twins: Self-Supervised Audio Representation Learningcited by 5
  14. 2023RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustnesscited by 5
  15. 2022DeLoRes: Decorrelating Latent Spaces for Low-Resource Audio Representation Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024An Efficient End-to-End Approach to Noise Invariant Speech Features via Multi-Task Learning
  2. 2022Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement
  3. 2025Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment
  4. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
  5. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  6. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  7. 2025Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance
  8. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR