mlsanityData to 7 Oct 2026

Autoregressive Knowledge Distillation through Imitation Learning

arXiv 2009.07253 ↗First posted 15 Sep 2020cs.CL cs.LGcited by 68

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering inference speed, making state-of-the-art models cumbersome to deploy in real-world, time-sensitive settings. We develop a compression technique for autoregressive models that is driven by an imitation learning perspective on knowledge distillation. The algorithm is designed to address the exposure bias problem. On prototypical language generation tasks such as translation and summarization, our method consistently outperforms other distillation algorithms, such as sequence-level knowledge distillation. Student models trained with our method attain 1.4 to 4.8 BLEU/ROUGE points higher than those trained from scratch, while increasing inference speed by up to 14 times in comparison to the teacher model.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 55 references that are in the map. First posted on arXiv 15 Sep 2020.

Builds on

43 of 55 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
31 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  3. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  5. 2016Sequence-Level Knowledge Distillationcited by 761
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  8. 2019Text Summarization with Pretrained Encoderscited by 553
  9. 2017Non-Autoregressive Neural Machine Translationcited by 457
  10. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  11. 2016Using the Output Embedding to Improve Language Modelscited by 266
  12. 2018Scaling Neural Machine Translationcited by 204
  13. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  14. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  15. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  16. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  17. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  18. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  19. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  20. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  21. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  22. 2015Learning to Search Better Than Your Teachercited by 50
  23. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  24. 2018Latent Alignment and Variational Attentioncited by 36
  25. 2019Imitation Learning for Non-Autoregressive Neural Machine Translationcited by 36
  26. 2017A Teacher-Student Framework for Zero-Resource Neural Machine Translationcited by 22
  27. 2017SEARNN: Training RNNs with Global-Local Lossescited by 17
  28. 2017A Copy-Augmented Sequence-to-Sequence Architecture Gives Good Performance on Task-Oriented Dialoguecited by 14
  29. 2017Differentiable Scheduled Sampling for Credit Assignmentcited by 13
  30. 2016Distilling an Ensemble of Greedy Dependency Parsers into One MST Parsercited by 11
  31. 2018Logarithmic Regret for Online Gradient Descent Beyond Strong Convexitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 0; 2023: 8; 2024: 14; 2025: 12; 2026: 30.

68 papers.

Sort:

By area:

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  3. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  5. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  6. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  7. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  8. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  9. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  10. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43
  11. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 41
  12. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
38 more
  1. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  2. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 21
  3. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  4. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  5. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  6. 2021Teaching Autoregressive Language Models Complex Tasks By Demonstrationcited by 15
  7. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillationcited by 13
  8. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  9. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  10. 2024Diversity-Rewarded CFG Distillationcited by 9
  11. 2026OVD: On-policy Verbal Distillationcited by 9
  12. 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Modelscited by 8
  13. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 7
  14. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 7
  15. 2024Generative Prompt Internalizationcited by 6
  16. 2025Revealing the Power of Post-Training for Small Language Models via Knowledge Distillationcited by 6
  17. 2026Rethinking Privileged Information in On-Policy Self-Distillationcited by 6
  18. 2023Target-Side Augmentation for Document-Level Machine Translationcited by 5
  19. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  20. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Modelscited by 5
  21. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  22. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 5
  23. 2021Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversationcited by 4
  24. 2025Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Frameworkcited by 4
  25. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4
  26. 2025AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switchingcited by 4
  27. 2026Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Modelscited by 4
  28. 2026Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridgingcited by 4
  29. 2023Enhancing Supervised Learning with Contrastive Markings in Neural Machine Translation Trainingcited by 3
  30. 2023Extending Multilingual Machine Translation through Imitation Learningcited by 3
  31. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  32. 2025Membership and Memorization in LLM Knowledge Distillationcited by 3
  33. 2024Multi-Granularity Semantic Revision for Large Language Model Distillationcited by 2
  34. 2024Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQLcited by 2
  35. 2025AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distributioncited by 2
  36. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  37. 2026Knowledge Distillation for Visual Autoregressive Modelscited by 2
  38. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
  2. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  3. 2025Knowledge Distillation with Training Wheels
  4. 2026A Survey of On-Policy Distillation for Large Language Models
  5. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
  6. 2025Language Model Distillation: A Temporal Difference Imitation Learning Perspective
  7. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  8. 2026SelFusion: Self-distillation for Diffusion Language Models