mlsanityData to 14 Oct 2026

Understanding and Improving Lexical Choice in Non-Autoregressive Translation

arXiv 2012.14583 ↗First posted 29 Dec 2020cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. The source code will be released.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 16; 2023: 15; 2024: 7; 2025: 2; 2026: 4.

56 papers.

Sort:

By area:

  1. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  2. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  3. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  4. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  5. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  6. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  7. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  8. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  9. 2022Learning Graph Neural Networks for Image Style Transfercited by 29
  10. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 29
  11. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 28
  12. 2022On the Learning of Non-Autoregressive Transformerscited by 27
38 more
  1. 2021Progressive Multi-Granularity Training for Non-Autoregressive Translationcited by 25
  2. 2023Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMscited by 25
  3. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  4. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Treecited by 20
  5. 2021Improving Neural Machine Translation by Bidirectional Trainingcited by 17
  6. 2022Pose-disentangled Contrastive Learning for Self-supervised Facial Representationcited by 17
  7. 2024WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledgecited by 16
  8. 2022Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translationcited by 14
  9. 2022Dynamic Contrastive Distillation for Image-Text Retrievalcited by 14
  10. 2022Improving Simultaneous Machine Translation with Monolingual Datacited by 13
  11. 2024Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extractioncited by 13
  12. 2021How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?cited by 12
  13. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  14. 2021The USYD-JD Speech Translation System for IWSLT 2021cited by 11
  15. 2021Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysiscited by 11
  16. 2023Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translationcited by 11
  17. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretrainingcited by 11
  18. 2021Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understandingcited by 9
  19. 2021Self-Guided Curriculum Learning for Neural Machine Translationcited by 9
  20. 2022Vega-MT: The JD Explore Academy Translation System for WMT22cited by 9
  21. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8
  22. 2023Self-Evolution Learning for Discriminative Language Model Pretrainingcited by 8
  23. 2022Multi-Granularity Optimization for Non-Autoregressive Translationcited by 6
  24. 2023Noise-Resistant Multimodal Transformer for Emotion Recognitioncited by 6
  25. 2026The Bitter Lesson of Diffusion Language Models for Agentic Workflows: A Comprehensive Reality Checkcited by 6
  26. 2022A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translationcited by 5
  27. 2023AMOM: Adaptive Masking over Masking for Conditional Masked Language Modelcited by 5
  28. 2023Prompt-Learning for Cross-Lingual Relation Extractioncited by 4
  29. 2021Rule-based Morphological Inflection Improves Neural Terminology Translationcited by 3
  30. 2021Variance-Aware Machine Translation Test Setscited by 3
  31. 2022Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraintscited by 3
  32. 2022Hybrid-Regressive Neural Machine Translationcited by 3
  33. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  34. 2022Original or Translated? On the Use of Parallel Data for Translation Quality Estimationcited by 2
  35. 2023OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML Systemcited by 2
  36. 2023Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translationcited by 2
  37. 2023Revisiting Non-Autoregressive Translation at Scalecited by 2
  38. 2023Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translationcited by 2

50 most cited of 56 All 56 (JSON)