# ENCONTER: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer

arXiv: https://arxiv.org/abs/2103.09548 · HTML: https://mlsanity.com/paper/2103.09548/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018)
- [CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling](https://mlsanity.com/paper/1811.10996/) (2018)
- [Target-Guided Open-Domain Conversation](https://mlsanity.com/paper/1905.11553/) (2019)
- [Non-Monotonic Sequential Text Generation](https://mlsanity.com/paper/1902.02192/) (2019)
- [RecipeGPT: Generative Pre-training Based Cooking Recipe Generation and Evaluation System](https://mlsanity.com/paper/2003.02498/) (2020)
- [Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading](https://mlsanity.com/paper/1906.02738/) (2019)
- [KERMIT: Generative Insertion-Based Modeling for Sequences](https://mlsanity.com/paper/1906.01604/) (2019)
- [COCO_TS Dataset: Pixel-level Annotations Based on Weak Supervision for Scene Text Segmentation](https://mlsanity.com/paper/1904.00818/) (2019)
- [POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training](https://mlsanity.com/paper/2005.00558/) (2020)

## Built on by (most cited first, 3 of 3)

- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [Monte Carlo Tree Search for Recipe Generation using GPT-2](https://mlsanity.com/paper/2401.05199/) (2024)
- [Incorporating Structure and Chord Constraints in Symbolic Transformer-based Melodic Harmonization](https://mlsanity.com/paper/2512.07627/) (2025)

## Abstract (arXiv)

Pretrained using large amount of data, autoregressive language models are able to generate high quality sequences. However, these models do not perform well under hard lexical constraints as they lack fine control of content generation process. Progressive insertion-based transformers can overcome the above limitation and efficiently generate a sequence in parallel given some input tokens as constraint. These transformers however may fail to support hard lexical constraints as their generation process is more likely to terminate prematurely. The paper analyses such early termination problems and proposes the Entity-constrained insertion transformer (ENCONTER), a new insertion transformer that addresses the above pitfall without compromising much generation efficiency. We introduce a new training strategy that considers predefined hard lexical constraints (e.g., entities to be included in the generated sequence). Our experiments show that ENCONTER outperforms other baseline models in several performance metrics rendering it more suitable in practical applications. Our code is available at https://github.com/LARC-CMU-SMU/Enconter