mlsanityData to 14 Oct 2026

GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model

arXiv 2406.09444 ↗First posted 12 Jun 2024eess.AS cs.CL cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-trained speech language models such as HuBERT and WavLM leverage unlabeled speech data for self-supervised learning and offer powerful representations for numerous downstream tasks. Despite the success of these models, their high requirements for memory and computing resource hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework which generates the hidden representations of the pre-trained teacher model directly by a much smaller student network. The proposed method takes the previous hidden layer as history and implements a layer-by-layer prediction of the teacher model autoregressively. Experiments on SUPERB reveal the advantage of GenDistiller over the baseline distilling method without an autoregressive framework, with 33% fewer parameters, similar time consumption and better performance on most of the SUPERB tasks. Ultimately, the proposed GenDistiller reduces the size of WavLM by 82%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

31 of 36 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
19 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  6. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  7. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  8. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  9. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  12. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  13. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57
  14. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  15. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  16. 2023Generic-to-Specific Distillation of Masked Autoencoderscited by 19
  17. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 15
  18. 2021KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillationcited by 11
  19. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compressioncited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023GenDistiller: Distilling Pre-trained Language Models based on Generative Models
  2. 2020Autoregressive Knowledge Distillation through Imitation Learning
  3. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  5. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
  6. 2024Revisiting Knowledge Distillation for Autoregressive Language Models
  7. 2023MiniLLM: On-Policy Distillation of Large Language Models
  8. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models