mlsanityData to 14 Oct 2026

Aligned Cross Entropy for Non-Autoregressive Machine Translation

arXiv 2004.01655 ↗First posted 3 Apr 2020cs.CL cs.LG stat.MLcited by 65

Not on the map. Why?

Abstract (arXiv)

Non-autoregressive machine translation models significantly speed up decoding by allowing for parallel prediction of the entire target sequence. However, modeling word order is more challenging due to the lack of autoregressive factors in the model. This difficultly is compounded during training with cross entropy loss, which can highly penalize small shifts in word order. In this paper, we propose aligned cross entropy (AXE) as an alternative loss function for training of non-autoregressive models. AXE uses a differentiable dynamic program to assign loss based on the best possible monotonic alignment between target tokens and model predictions. AXE-based training of conditional masked language models (CMLMs) substantially improves performance on major WMT benchmarks, while setting a new state of the art for non-autoregressive models.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 21; 2023: 16; 2024: 6; 2025: 2; 2026: 2.

65 papers.

Sort:

By area:

  1. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  2. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  3. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  4. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  5. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  6. 2021FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognitioncited by 43
  7. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  8. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  9. 2022On the Learning of Non-Autoregressive Transformerscited by 27
  10. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  11. 2021AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translatecited by 24
  12. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
38 more
  1. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  2. 2021O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioningcited by 18
  3. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  4. 2021Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialogcited by 17
  5. 2021Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generationcited by 17
  6. 2022Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translationcited by 16
  7. 2022Chaining Simultaneous Thoughts for Numerical Reasoningcited by 14
  8. 2021Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translationcited by 13
  9. 2022Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 13
  10. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  11. 2023Non-autoregressive Streaming Transformer for Simultaneous Translationcited by 12
  12. 2022Non-Autoregressive Machine Translation: It's Not as Fast as it Seemscited by 11
  13. 2023Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translationcited by 11
  14. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  15. 2022LegoNN: Building Modular Encoder-Decoder Modelscited by 9
  16. 2021Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translationcited by 8
  17. 2022ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 8
  18. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8
  19. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  20. 2022Multi-Granularity Optimization for Non-Autoregressive Translationcited by 6
  21. 2021Non-autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognitioncited by 5
  22. 2022Non-Autoregressive Neural Machine Translation: A Call for Claritycited by 5
  23. 2022A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translationcited by 5
  24. 2023AMOM: Adaptive Masking over Masking for Conditional Masked Language Modelcited by 5
  25. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  26. 2023Non-autoregressive Machine Translation with Probabilistic Context-free Grammarcited by 4
  27. 2025Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencingcited by 4
  28. 2021Can Latent Alignments Improve Autoregressive Machine Translation?cited by 3
  29. 2022Hybrid-Regressive Neural Machine Translationcited by 3
  30. 2022Rephrasing the Reference for Non-Autoregressive Machine Translationcited by 3
  31. 2022Diffusion Glancing Transformer for Parallel Sequence to Sequence Learningcited by 3
  32. 2021Diformer: Directional Transformer for Neural Machine Translationcited by 2
  33. 2023Dynamic Alignment Mask CTC: Improved Mask-CTC with Aligned Cross Entropycited by 2
  34. 2023Revisiting Non-Autoregressive Translation at Scalecited by 2
  35. 2024Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrievalcited by 2
  36. 2021Can Multilinguality benefit Non-autoregressive Machine Translation?cited by 1
  37. 2022Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translatorscited by 1
  38. 2023RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformercited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Modeling Coverage for Non-Autoregressive Neural Machine Translation
  2. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  3. 2022CUNI Non-Autoregressive System for the WMT 22 Efficient Translation Shared Task
  4. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
  5. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translation
  6. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding
  7. 2022A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
  8. 2025Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference