mlsanityData to 14 Oct 2026

GenDistiller: Distilling Pre-trained Language Models based on Generative Models

arXiv 2310.13418 ↗First posted 20 Oct 2023eess.AS eess.SPcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Self-supervised pre-trained models such as HuBERT and WavLM leverage unlabeled speech data for representation learning and offer significantly improve for numerous downstream tasks. Despite the success of these methods, their large memory and strong computational requirements hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework to distill hidden representations from teacher network based on generative language model. The generative structure enables the proposed model to generate the target teacher hidden layers autoregressively, considering the interactions between hidden layers without instroducing additional inputs. A two-dimensional attention mechanism is implemented to ensure the causality of hidden layers, while preserving bidirectional attention in the time dimension. Experiments reveal the advantage of the generative distiller over the baseline system that predicts the hidden layers of teacher network directly without a generatvie model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

33 of 40 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
21 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  8. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  9. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  10. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  11. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  12. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  13. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  14. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  15. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57
  16. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  17. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  18. 2023Generic-to-Specific Distillation of Masked Autoencoderscited by 19
  19. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 15
  20. 2021KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillationcited by 11
  21. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compressioncited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model
  2. 2026A Study on Hidden Layer Distillation for Large Language Model Pre-Training
  3. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
  4. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation
  5. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models
  6. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers
  7. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training
  8. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models