mlsanityData to 7 Oct 2026

BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining

arXiv 2012.15525 ↗First posted 31 Dec 2020cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation by designing a novel model structure for large-scale pretraining. The pretrained BANG model can simultaneously support AR, NAR and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum) and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39 and 5.90 in the overall scores of SQuAD, XSUM and PersonaChat respectively compared with the strong NAR baselines.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 10; 2023: 12; 2024: 4; 2025: 3.

33 papers.

Sort:

By area:

  1. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  2. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  3. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  4. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
  5. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  6. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  7. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  8. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 43
  9. 2021Paradigm Shift in Natural Language Processingcited by 39
  10. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 22
  11. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21
  12. 2022A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarizationcited by 13
21 more
  1. 2022Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarizationcited by 12
  2. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10
  3. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8
  4. 2024INSIGHT: Universal Neural Simulator for Analog Circuits Harnessing Autoregressive Transformerscited by 8
  5. 2021Improving Non-autoregressive Generation with Mixup Trainingcited by 7
  6. 2021EL-Attention: Memory Efficient Lossless Attention for Generationcited by 6
  7. 2023AMOM: Adaptive Masking over Masking for Conditional Masked Language Modelcited by 5
  8. 2023A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognitioncited by 5
  9. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  10. 2022Replacing Language Model for Style Transfercited by 3
  11. 2021Diformer: Directional Transformer for Neural Machine Translationcited by 2
  12. 2024Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrievalcited by 2
  13. 2022A Self-Paced Mixed Distillation Method for Non-Autoregressive Generationcited by 1
  14. 2023Optimizing Non-Autoregressive Transformers with Contrastive Learningcited by 1
  15. 2024Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Maskcited by 1
  16. 2025Multilingual Non-Autoregressive Machine Translation without Knowledge Distillationcited by 1
  17. 2025Pipelined Decoder for Efficient Context-Aware Text Generationcited by 1
  18. 2025Why Stop at Words? Unveiling the Bigger Picture through Line-Level OCRcited by 1
  19. 2023Non-Autoregressive Sentence Orderingcited by 0
  20. 2023UT5: Pretraining Non autoregressive T5 with unrolled denoisingcited by 0
  21. 2024Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translationcited by 0