# A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models

arXiv: https://arxiv.org/abs/1905.12790 · HTML: https://mlsanity.com/paper/1905.12790/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 17 of 17)

- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings](https://mlsanity.com/paper/2106.02736/) (2021)
- [Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning](https://mlsanity.com/paper/2203.02573/) (2022)
- [M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers](https://mlsanity.com/paper/2105.14211/) (2021)
- [UniMASK: Unified Inference in Sequential Decision Problems](https://mlsanity.com/paper/2211.10869/) (2022)
- [Arrows of Time for Large Language Models](https://mlsanity.com/paper/2401.17505/) (2024)
- [COFS: Controllable Furniture layout Synthesis](https://mlsanity.com/paper/2205.14657/) (2022)
- [Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers](https://mlsanity.com/paper/2204.13326/) (2022)
- [Twist Decoding: Diverse Generators Guide Each Other](https://mlsanity.com/paper/2205.09273/) (2022)
- [Causal Scene BERT: Improving object detection by searching for challenging groups of data](https://mlsanity.com/paper/2202.03651/) (2022)
- [M6-Fashion: High-Fidelity Multi-modal Image Generation and Editing](https://mlsanity.com/paper/2205.11705/) (2022)
- [Diformer: Directional Transformer for Neural Machine Translation](https://mlsanity.com/paper/2112.11632/) (2021)
- [Why Stop at Words? Unveiling the Bigger Picture through Line-Level OCR](https://mlsanity.com/paper/2508.21693/) (2025)
- [Learning and Analyzing Generation Order for Undirected Sequence Models](https://mlsanity.com/paper/2112.09097/) (2021)
- [P3LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training](https://mlsanity.com/paper/2210.12339/) (2022)
- [Towards Computationally Verifiable Semantic Grounding for Language Models](https://mlsanity.com/paper/2211.09070/) (2022)
- [Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation](https://mlsanity.com/paper/2402.18428/) (2024)

## Abstract (arXiv)

Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relatively little attention, in part because generating from undirected models departs significantly from conventional monotonic generation in directed sequence models. We investigate this problem by proposing a generalized model of sequence generation that unifies decoding in directed and undirected models. The proposed framework models the process of generation rather than the resulting sequence, and under this framework, we derive various neural sequence models as special cases, such as autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. This unification enables us to adapt decoding algorithms originally developed for directed sequence models to undirected sequence models. We demonstrate this by evaluating various handcrafted and learned decoding strategies on a BERT-like machine translation model (Lample & Conneau, 2019). The proposed approach achieves constant-time translation results on par with linear-time translation results from the same undirected sequence model, while both are competitive with the state-of-the-art on WMT'14 English-German translation.