# Directed Acyclic Transformer for Non-Autoregressive Machine Translation

arXiv: https://arxiv.org/abs/2205.07459 · HTML: https://mlsanity.com/paper/2205.07459/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 16 May 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [Mixture Models for Diverse Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/1902.07816/) (2019)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018)
- [Aligned Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2004.01655/) (2020)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019)
- [Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision](https://mlsanity.com/paper/2110.07515/) (2021)
- [Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input](https://mlsanity.com/paper/1812.09664/) (2018)
- [Imitation Learning for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1906.02041/) (2019)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [Semi-Autoregressive Training Improves Mask-Predict Decoding](https://mlsanity.com/paper/2001.08785/) (2020)
- [First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs](https://mlsanity.com/paper/1408.2873/) (2014)
- [Non-Autoregressive Translation by Learning Target Categorical Codes](https://mlsanity.com/paper/2103.11405/) (2021)

## Built on by (most cited first, 30 of 44)

- [Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference](https://mlsanity.com/paper/2508.02193/) (2025)
- [Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning](https://mlsanity.com/paper/2410.14157/) (2024)
- [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://mlsanity.com/paper/2302.10025/) (2023)
- [Empowering Diffusion Models on the Embedding Space for Text Generation](https://mlsanity.com/paper/2212.09412/) (2022)
- [Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning](https://mlsanity.com/paper/2308.12219/) (2023)
- [On the Learning of Non-Autoregressive Transformers](https://mlsanity.com/paper/2206.05975/) (2022)
- [Chaining Simultaneous Thoughts for Numerical Reasoning](https://mlsanity.com/paper/2211.16482/) (2022)
- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)
- [BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models](https://mlsanity.com/paper/2401.12522/) (2024)
- [Non-autoregressive Streaming Transformer for Simultaneous Translation](https://mlsanity.com/paper/2310.14883/) (2023)
- [Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2303.06662/) (2023)
- [A Theory for Length Generalization in Learning to Reason](https://mlsanity.com/paper/2404.00560/) (2024)
- [A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2406.06937/) (2024)
- [Continuous Autoregressive Language Models](https://mlsanity.com/paper/2510.27688/) (2025)
- [DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation](https://mlsanity.com/paper/2310.07403/) (2023)
- [Simultaneous Machine Translation with Tailored Reference](https://mlsanity.com/paper/2310.13588/) (2023)
- [Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation](https://mlsanity.com/paper/2304.11791/) (2023)
- [Non-autoregressive Text Editing with Copy-aware Latent Alignments](https://mlsanity.com/paper/2310.07821/) (2023)
- [Non-Autoregressive Math Word Problem Solver with Unified Tree Structure](https://mlsanity.com/paper/2305.04556/) (2023)
- [Beyond Next Token Prediction: Patch-Level Training for Large Language Models](https://mlsanity.com/paper/2407.12665/) (2024)
- [AMOM: Adaptive Masking over Masking for Conditional Masked Language Model](https://mlsanity.com/paper/2303.07457/) (2023)
- [Non-autoregressive Machine Translation with Probabilistic Context-free Grammar](https://mlsanity.com/paper/2311.07941/) (2023)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning](https://mlsanity.com/paper/2212.10240/) (2022)
- [Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2303.17910/) (2023)
- [ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models](https://mlsanity.com/paper/2501.13397/) (2025)
- [DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models](https://mlsanity.com/paper/2603.15340/) (2026)
- [Shared Latent Space by Both Languages in Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2305.03511/) (2023)
- [Non-Autoregressive Document-Level Machine Translation](https://mlsanity.com/paper/2305.12878/) (2023)

## Abstract (arXiv)

Non-autoregressive Transformers (NATs) significantly reduce the decoding latency by generating all tokens in parallel. However, such independent predictions prevent NATs from capturing the dependencies between the tokens for generating multiple possible translations. In this paper, we propose Directed Acyclic Transfomer (DA-Transformer), which represents the hidden states in a Directed Acyclic Graph (DAG), where each path of the DAG corresponds to a specific translation. The whole DAG simultaneously captures multiple translations and facilitates fast predictions in a non-autoregressive fashion. Experiments on the raw training data of WMT benchmark show that DA-Transformer substantially outperforms previous NATs by about 3 BLEU on average, which is the first NAT model that achieves competitive results with autoregressive Transformers without relying on knowledge distillation.