mlsanityData to 7 Oct 2026

STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models

arXiv 2312.09040 ↗First posted 14 Dec 2023cs.SD cs.CL eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Albeit great performance of Transformer-based speech selfsupervised learning (SSL) models, their large parameter size and computational cost make them unfavorable to utilize. In this study, we propose to compress the speech SSL models by distilling speech temporal relation (STaR). Unlike previous works that directly match the representation for each speech frame, STaR distillation transfers temporal relation between speech frames, which is more suitable for lightweight student with limited capacity. We explore three STaR distillation objectives and select the best combination as the final STaR loss. Our model distilled from HuBERT BASE achieves an overall score of 79.8 on SUPERB benchmark, the best performance among models with up to 27 million parameters. We show that our method is applicable across different speech SSL models and maintains robust performance with further reduced parameters.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

22 of 35 references.

References · 22

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  7. 2019Correlation Congruence for Knowledge Distillationcited by 280
  8. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  9. 2021Unsupervised Speech Recognitioncited by 147
  10. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  11. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  12. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
10 more
  1. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  2. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  3. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  4. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  5. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  6. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 13
  7. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  8. 2021Speaker embeddings by modeling channel-wise correlationscited by 1
  9. 2023Lightweight feature encoder for wake-up word detection based on self-supervised speech representationcited by 1
  10. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verificationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 2
  2. 2025HuBERT-VIC: Improving Noise-Robust Automatic Speech Recognition of Speech Foundation Model via Variance-Invariance-Covariance Regularizationcited by 1
  3. 2026Fast Speech Foundation Model Distillation Using Interleaved Stackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding ↗
  2. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
  3. 2025An Exploration of Mamba for Speech Self-Supervised Models
  4. 2025LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
  5. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Models
  6. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognition
  7. 2025TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
  8. 2022Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation