# BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining

arXiv: https://arxiv.org/abs/2012.15525 · HTML: https://mlsanity.com/paper/2012.15525/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 30 of 33)

- [SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control](https://mlsanity.com/paper/2210.17432/) (2022)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation](https://mlsanity.com/paper/2305.09515/) (2023)
- [Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise](https://mlsanity.com/paper/2212.11685/) (2022)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://mlsanity.com/paper/2302.10025/) (2023)
- [Diffusion Models for Non-autoregressive Text Generation: A Survey](https://mlsanity.com/paper/2303.06574/) (2023)
- [Paradigm Shift in Natural Language Processing](https://mlsanity.com/paper/2109.12575/) (2021)
- [Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond](https://mlsanity.com/paper/2310.05317/) (2023)
- [Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2305.04044/) (2023)
- [A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization](https://mlsanity.com/paper/2205.14522/) (2022)
- [Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2205.14521/) (2022)
- [ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation](https://mlsanity.com/paper/2210.13304/) (2022)
- [Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation](https://mlsanity.com/paper/2304.11791/) (2023)
- [INSIGHT: Universal Neural Simulator for Analog Circuits Harnessing Autoregressive Transformers](https://mlsanity.com/paper/2407.07346/) (2024)
- [Improving Non-autoregressive Generation with Mixup Training](https://mlsanity.com/paper/2110.11115/) (2021)
- [EL-Attention: Memory Efficient Lossless Attention for Generation](https://mlsanity.com/paper/2105.04779/) (2021)
- [AMOM: Adaptive Masking over Masking for Conditional Masked Language Model](https://mlsanity.com/paper/2303.07457/) (2023)
- [A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition](https://mlsanity.com/paper/2304.07611/) (2023)
- [Beyond MLE: Convex Learning for Text Generation](https://mlsanity.com/paper/2310.17217/) (2023)
- [Replacing Language Model for Style Transfer](https://mlsanity.com/paper/2211.07343/) (2022)
- [Diformer: Directional Transformer for Neural Machine Translation](https://mlsanity.com/paper/2112.11632/) (2021)
- [Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrieval](https://mlsanity.com/paper/2406.06739/) (2024)
- [A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation](https://mlsanity.com/paper/2205.11162/) (2022)
- [Optimizing Non-Autoregressive Transformers with Contrastive Learning](https://mlsanity.com/paper/2305.13667/) (2023)
- [Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Mask](https://mlsanity.com/paper/2406.10034/) (2024)
- [Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation](https://mlsanity.com/paper/2502.04537/) (2025)
- [Pipelined Decoder for Efficient Context-Aware Text Generation](https://mlsanity.com/paper/2506.23431/) (2025)
- [Why Stop at Words? Unveiling the Bigger Picture through Line-Level OCR](https://mlsanity.com/paper/2508.21693/) (2025)

## Abstract (arXiv)

In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation by designing a novel model structure for large-scale pretraining. The pretrained BANG model can simultaneously support AR, NAR and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum) and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39 and 5.90 in the overall scores of SQuAD, XSUM and PersonaChat respectively compared with the strong NAR baselines.