mlsanityData to 14 Oct 2026

DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation

arXiv 2206.02999 ↗First posted 7 Jun 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The computational benefits of iterative non-autoregressive transformers decrease as the number of decoding steps increases. As a remedy, we introduce Distill Multiple Steps (DiMS), a simple yet effective distillation technique to decrease the number of required steps to reach a certain translation quality. The distilled model enjoys the computational benefits of early iterations while preserving the enhancements from several iterative steps. DiMS relies on two models namely student and teacher. The student is optimized to predict the output of the teacher after multiple decoding steps while the teacher follows the student via a slow-moving average. The moving average keeps the teacher's knowledge updated and enhances the quality of the labels provided by the teacher. During inference, the student is used for translation and no additional computation is added. We verify the effectiveness of DiMS on various models obtaining 7.8 and 12.9 BLEU points improvements in single-step translation accuracy on distilled and raw versions of WMT'14 De-En.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

32 of 42 references.

References · 32

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  11. 2017Non-Autoregressive Neural Machine Translationcited by 463
  12. 2019Levenshtein Transformercited by 199
20 more
  1. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  2. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 125
  3. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 122
  4. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  5. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  6. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  7. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  8. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  9. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  10. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translationcited by 65
  11. 2019Fast Structured Decoding for Sequence Modelscited by 59
  12. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  13. 2019Non-Autoregressive Machine Translation with Auxiliary Regularizationcited by 52
  14. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  15. 2019Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translationcited by 40
  16. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  17. 2020Semi-Autoregressive Training Improves Mask-Predict Decodingcited by 33
  18. 2021AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translatecited by 24
  19. 2020An EM Approach to Non-autoregressive Conditional Sequence Generationcited by 21
  20. 2020Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translationcited by 13

Built on by

Not cited yet.