mlsanityData to 7 Oct 2026

Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation

arXiv 2305.11685 ↗First posted 19 May 2023eess.AS cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Transformer-based speech self-supervised learning (SSL) models, such as HuBERT, show surprising performance in various speech processing tasks. However, huge number of parameters in speech SSL models necessitate the compression to a more compact model for wider usage in academia or small companies. In this study, we suggest to reuse attention maps across the Transformer layers, so as to remove key and query parameters while retaining the number of layers. Furthermore, we propose a novel masking distillation strategy to improve the student model's speech representation quality. We extend the distillation loss to utilize both masked and unmasked speech frames to fully leverage the teacher model's high-quality representation. Our universal compression strategy yields the student model that achieves phoneme error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB benchmark.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

29 of 32 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
17 more
  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  4. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  5. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  6. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  7. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  8. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  9. 2022Audio Self-supervised Learning: A Surveycited by 65
  10. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 58
  11. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  12. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  13. 2019Sharing Attention Weights for Fast Transformercited by 34
  14. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  15. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  16. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  17. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 2; 2026: 1.

6 papers.

  1. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  2. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 4
  3. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  4. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 2
  5. 2025Adaptive Knowledge Distillation for Device-Directed Speech Detectioncited by 2
  6. 2026Fast Speech Foundation Model Distillation Using Interleaved Stackingcited by 0