mlsanityData to 7 Oct 2026

Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation

arXiv 1912.03334 ↗First posted 6 Dec 2019cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Sequence-level knowledge distillation (SLKD) is a model compression technique that leverages large, accurate teacher models to train smaller, under-parameterized student models. Why does pre-processing MT data with SLKD help us train smaller models? We test the common hypothesis that SLKD addresses a capacity deficiency in students by "simplifying" noisy data points and find it unlikely in our case. Models trained on concatenations of original and "simplified" datasets generalize just as well as baseline SLKD. We then propose an alternative hypothesis under the lens of data augmentation and regularization. We try various augmentation strategies and observe that dropout regularization can become unnecessary. Our methods achieve BLEU gains of 0.7-1.2 on TED Talks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2022Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generationcited by 67
  2. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  3. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  4. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  5. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  6. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  7. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  8. 2022A baseline revisited: Pushing the limits of multi-segment models for context-aware translationcited by 11
  9. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  10. 2023Target-Side Augmentation for Document-Level Machine Translationcited by 5
  11. 2023Heterogeneous Generative Knowledge Distillation with Masked Image Modelingcited by 2
  12. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
4 more
  1. 2025Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translationcited by 2
  2. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  3. 2021ESPnet-ST IWSLT 2021 Offline Speech Translation Systemcited by 0
  4. 2023Unraveling Key Factors of Knowledge Distillationcited by 0