mlsanityData to 14 Oct 2026

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation

arXiv 2304.11791 ↗First posted 24 Apr 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation. However, in a wider range of text generation tasks, existing NAR models lack proper pre-training, making them still far behind the pre-trained autoregressive models. In this paper, we propose Pre-trained Directed Acyclic Transformer (PreDAT) and a novel pre-training task to promote prediction consistency in NAR generation. Experiments on five text generation tasks show that our PreDAT remarkably outperforms existing pre-trained NAR models (+4.2 scores on average) and even achieves better results than pre-trained autoregressive baselines in n-gram-based metrics, along with 17 times speedup in throughput. Further analysis shows that PreDAT benefits from the unbiased prediction order that alleviates the error accumulation problem in autoregressive generation, which provides new insights into the advantages of NAR generation.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

54 of 76 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
38 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  2. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  3. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  5. 2016Sequence-Level Knowledge Distillationcited by 801
  6. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  7. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  8. 2017Non-Autoregressive Neural Machine Translationcited by 463
  9. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  10. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  12. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  13. 2019Levenshtein Transformercited by 199
  14. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  15. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  16. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  17. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  18. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  19. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  20. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  21. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  22. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translationcited by 65
  23. 2019Fast Structured Decoding for Sequence Modelscited by 59
  24. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  25. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 56
  26. 2020Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraintscited by 48
  27. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  28. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  29. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  30. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  31. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  32. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  33. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
  34. 2019Hint-Based Training for Non-Autoregressive Machine Translationcited by 28
  35. 2022On the Learning of Non-Autoregressive Transformerscited by 27
  36. 2022textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generationcited by 20
  37. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  38. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 1; 2026: 2.

8 papers.

  1. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  2. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  3. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  4. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  5. 2026DOS: Dependency-Oriented Sampler for Masked Diffusion Language Modelscited by 2
  6. 2023Optimizing Non-Autoregressive Transformers with Contrastive Learningcited by 1
  7. 2024Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automatacited by 1
  8. 2026DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Modelscited by 0