mlsanityData to 14 Oct 2026

Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models

arXiv 2207.06867 ↗First posted 14 Jul 2022cs.CL cs.SD eess.AScited by 31

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Self-supervised learning (SSL) is seen as a very promising approach with high performance for several speech downstream tasks. Since the parameters of SSL models are generally so large that training and inference require a lot of memory and computational cost, it is desirable to produce compact SSL models without a significant performance degradation by applying compression methods such as knowledge distillation (KD). Although the KD approach is able to shrink the depth and/or width of SSL model structures, there has been little research on how varying the depth and width impacts the internal representation of the small-footprint model. This paper provides an empirical study that addresses the question. We investigate the performance on SUPERB while varying the structure and KD methods so as to keep the number of parameters constant; this allows us to analyze the contribution of the representation introduced by varying the model architecture. Experiments demonstrate that a certain depth is essential for solving content-oriented tasks (e.g. automatic speech recognition) accurately, whereas a certain width is necessary for achieving high performance on several speaker-oriented tasks (e.g. speaker identification). Based on these observations, we identify, for SUPERB, a more compressed model with better performance than previous studies.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 36 of its 41 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

36 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models7 Oct

Other references · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
23 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  3. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  4. 2013Do Deep Nets Really Need to be Deep?cited by 474
  5. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  6. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  7. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  8. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  9. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  10. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  12. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  13. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  14. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  15. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 66
  16. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  17. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  18. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  19. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  20. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  21. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 21
  22. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
  23. 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 14; 2024: 5; 2025: 7; 2026: 3.

31 papers.

Sort:

By area:

  1. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  2. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  3. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  4. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 18
  5. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 12
  6. 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 11
  7. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  8. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  9. 2025Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Modelscited by 7
  10. 2022Enhancing and Adversarial: Improve ASR with Speaker Labelscited by 6
  11. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  12. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 5
19 more
  1. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  2. 2023Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Modelscited by 4
  3. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
  4. 2024Sustainable self-supervised learning for speech representationscited by 4
  5. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  6. 2022Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 3
  7. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  8. 2024Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Modelscited by 3
  9. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  10. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  11. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  12. 2025HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 1
  13. 2023A Small and Fast BERT for Chinese Medical Punctuation Restorationcited by 0
  14. 2023Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiencycited by 0
  15. 2025Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Modelscited by 0
  16. 2025Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPscited by 0
  17. 2026HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 0
  18. 2026Fast Speech Foundation Model Distillation Using Interleaved Stackingcited by 0
  19. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
  2. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
  3. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  4. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  5. 2022Application of Knowledge Distillation to Multi-task Speech Representation Learning
  6. 2022Improving generalizability of distilled self-supervised speech processing models under distorted settings
  7. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning
  8. 2022Label-free Knowledge Distillation with Contrastive Loss for Light-weight Speaker Recognition