mlsanityData to 14 Oct 2026

On the Learning of Non-Autoregressive Transformers

arXiv 2206.05975 ↗First posted 13 Jun 2022cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Non-autoregressive Transformer (NAT) is a family of text generation models, which aims to reduce the decoding latency by predicting the whole sentences in parallel. However, such latency reduction sacrifices the ability to capture left-to-right dependencies, thereby making NAT learning very challenging. In this paper, we present theoretical and empirical analyses to reveal the challenges of NAT learning and propose a unified perspective to understand existing successes. First, we show that simply training NAT by maximizing the likelihood can lead to an approximation of marginal distributions but drops all dependencies between tokens, where the dropped information can be measured by the dataset's conditional total correlation. Second, we formalize many previous objectives in a unified framework and show that their success can be concluded as maximizing the likelihood on a proxy distribution, leading to a reduced information loss. Empirical studies show that our perspective can explain the phenomena in NAT learning and guide the design of new training methods.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

35 of 47 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  4. 2016Sequence-Level Knowledge Distillationcited by 801
  5. 2017Non-Autoregressive Neural Machine Translationcited by 463
  6. 2019Levenshtein Transformercited by 199
  7. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  8. 2020A Theory of Usable Information Under Computational Constraintscited by 162
  9. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  10. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 122
  11. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  12. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
23 more
  1. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 104
  2. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  3. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  4. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  5. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  6. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  7. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translationcited by 65
  8. 2019Fast Structured Decoding for Sequence Modelscited by 59
  9. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  10. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  11. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 56
  12. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  13. 2019Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translationcited by 40
  14. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  15. 2019Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Informationcited by 37
  16. 2019Imitation Learning for Non-Autoregressive Neural Machine Translationcited by 36
  17. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  18. 2021Non-Autoregressive Translation by Learning Target Categorical Codescited by 25
  19. 2020An EM Approach to Non-autoregressive Conditional Sequence Generationcited by 21
  20. 2022textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generationcited by 20
  21. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  22. 2019Non-autoregressive Transformer by Position Learningcited by 17
  23. 2021The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21cited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 6; 2025: 3; 2026: 9.

27 papers.

Sort:

By area:

  1. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  2. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 64
  3. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  4. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  5. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 43
  6. 2022Transformers over Directed Acyclic Graphscited by 23
  7. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 18
  8. 2023Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translationcited by 11
  9. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  10. 2022ECM-OPCC: Efficient Context Model for Octree-based Point Cloud Compressioncited by 8
  11. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8
  12. 2023Non-autoregressive Text Editing with Copy-aware Latent Alignmentscited by 8
15 more
  1. 2024Modeling Temporal Dependencies within the Target for Long-Term Time Series Forecastingcited by 6
  2. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  3. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 3
  4. 2026FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Versioncited by 2
  5. 2024What Have We Achieved on Non-autoregressive Translation?cited by 1
  6. 2023Unlocking the Power of GANs in Non-Autoregressive Text Generationcited by 0
  7. 2024On the Information Redundancy in Non-Autoregressive Translationcited by 0
  8. 2024Non-autoregressive Personalized Bundle Generationcited by 0
  9. 2026Sentence Curve Language Modelscited by 0
  10. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  11. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  12. 2026DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Modelscited by 0
  13. 2026Decentralized Master-Mind: Joint Action Refinement through Iterative Intent Denoising in Multi-Agent Pathfindingcited by 0
  14. 2026Alpha Diffusion Language Models: Factorization Alone Is Not the Problemcited by 0
  15. 2026D-Loop: Looped Diffusion Drafting for Speculative Decodingcited by 0