mlsanityData to 7 Oct 2026

A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training

arXiv 2305.02031 ↗First posted 3 May 2023cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Modern Natural Language Generation (NLG) models come with massive computational and storage requirements. In this work, we study the potential of compressing them, which is crucial for real-world applications serving millions of users. We focus on Knowledge Distillation (KD) techniques, in which a small student model learns to imitate a large teacher model, allowing to transfer knowledge from the teacher to the student. In contrast to much of the previous work, our goal is to optimize the model for a specific NLG task and a specific dataset. Typically in real-world applications, in addition to labeled data there is abundant unlabeled task-specific data, which is crucial for attaining high compression rates via KD. In this work, we conduct a systematic study of task-specific KD techniques for various NLG tasks under realistic assumptions. We discuss the special characteristics of NLG distillation and particularly the exposure bias problem. Following, we derive a family of Pseudo-Target (PT) augmentation methods, substantially extending prior work on sequence-level KD. We propose the Joint-Teaching method, which applies word-level KD to multiple PTs generated by both the teacher and the student. Finally, we validate our findings in an extreme setup with no labeled examples using GPT-4 as the teacher. Our study provides practical model design observations and demonstrates the effectiveness of PT training for task-specific KD in NLG.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 57 of its 85 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

57 of 85 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  8. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  9. 2019When Does Label Smoothing Help?cited by 985
  10. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  13. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  14. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  15. 2019Abductive Commonsense Reasoningcited by 300
  16. 2022UL2: Unifying Language Learning Paradigmscited by 292
  17. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  18. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  19. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  20. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
  21. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  22. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  23. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  24. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  25. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  26. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  27. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  28. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  29. 2020Pre-trained Summarization Distillationcited by 70
  30. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  31. 2017Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Modelscited by 57
  32. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  33. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  34. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 36
  35. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  36. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  37. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  38. 2021Relating Neural Text Degeneration to Exposure Biascited by 19

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  2. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  3. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  4. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  5. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  6. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  7. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  8. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  9. 2023Measuring the Robustness of NLP Models to Domain Shiftscited by 11
  10. 2024AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillationcited by 5
  11. 2024BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLMcited by 5
  12. 2024DiDOTS: Knowledge Distillation from Large-Language-Models for Dementia Obfuscation in Transcribed Speechcited by 3
4 more
  1. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  2. 2025MiRAGE: Misconception Detection with Retrieval-Guided Multi-Stage Reasoning and Ensemble Fusioncited by 1
  3. 2025StatLLaMA: Multi-Stage training for domain-optimized statistical large language modelscited by 0
  4. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0