# Parallel Refinements for Lexically Constrained Text Generation with BART

arXiv: https://arxiv.org/abs/2109.12487 · HTML: https://mlsanity.com/paper/2109.12487/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Learning to Write with Cooperative Discriminators](https://mlsanity.com/paper/1805.06087/) (2018)
- [CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling](https://mlsanity.com/paper/1811.10996/) (2018)
- [Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach](https://mlsanity.com/paper/1805.05181/) (2018)
- [Lexically Constrained Neural Machine Translation with Levenshtein Transformer](https://mlsanity.com/paper/2004.12681/) (2020)
- [Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation](https://mlsanity.com/paper/1607.00970/) (2016)
- [Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series](https://mlsanity.com/paper/1504.01575/) (2015)
- [Progressive Generation of Long Text with Pretrained Language Models](https://mlsanity.com/paper/2006.15720/) (2020)
- [Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet](https://mlsanity.com/paper/2109.05797/) (2021)
- [Backward and Forward Language Modeling for Constrained Sentence Generation](https://mlsanity.com/paper/1512.06612/) (2015)
- [BFGAN: Backward and Forward Generative Adversarial Networks for Lexically Constrained Sentence Generation](https://mlsanity.com/paper/1806.08097/) (2018)
- [Incorporating Discriminator in Sentence Generation: a Gibbs Sampling Method](https://mlsanity.com/paper/1802.08970/) (2018)

## Built on by (most cited first, 24 of 24)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [MeaCap: Memory-Augmented Zero-shot Image Captioning](https://mlsanity.com/paper/2403.03715/) (2024)
- [Flexible-length Text Infilling for Discrete Diffusion Models](https://mlsanity.com/paper/2506.13579/) (2025)
- [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://mlsanity.com/paper/2211.10330/) (2022)
- [An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation](https://mlsanity.com/paper/2212.09387/) (2022)
- [Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning](https://mlsanity.com/paper/2210.11708/) (2022)
- [Toward Unified Controllable Text Generation via Regular Expression Instruction](https://mlsanity.com/paper/2309.10447/) (2023)
- [UCEpic: Unifying Aspect Planning and Lexical Constraints for Generating Explanations in Recommendation](https://mlsanity.com/paper/2209.13885/) (2022)
- [AutoTemplate: A Simple Recipe for Lexically Constrained Text Generation](https://mlsanity.com/paper/2211.08387/) (2022)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning](https://mlsanity.com/paper/2406.14164/) (2024)
- [LaVCa: LLM-assisted Visual Cortex Captioning](https://mlsanity.com/paper/2502.13606/) (2025)
- [Is it Possible to Modify Text to a Target Readability Level? An Initial Investigation Using Zero-Shot Large Language Models](https://mlsanity.com/paper/2309.12551/) (2023)
- [Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights](https://mlsanity.com/paper/2407.11449/) (2024)
- [Lexical Complexity Controlled Sentence Generation](https://mlsanity.com/paper/2211.14540/) (2022)
- [LLM Braces: Straightening Out LLM Predictions with Relevant Sub-Updates](https://mlsanity.com/paper/2503.16334/) (2025)
- [Incorporating Structure and Chord Constraints in Symbolic Transformer-based Melodic Harmonization](https://mlsanity.com/paper/2512.07627/) (2025)
- [How to Describe Images in a More Funny Way? Towards a Modular Approach to Cross-Modal Sarcasm Generation](https://mlsanity.com/paper/2211.10992/) (2022)
- [A Benchmark for Text Expansion: Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2309.09198/) (2023)
- [How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey](https://mlsanity.com/paper/2412.08158/) (2024)
- [The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning](https://mlsanity.com/paper/2503.23679/) (2025)
- [HiCaM: A Hierarchical-Causal Modification Framework for Long-Form Text Modification](https://mlsanity.com/paper/2505.24319/) (2025)
- [Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning](https://mlsanity.com/paper/2602.23588/) (2026)
- [A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principles](https://mlsanity.com/paper/2605.12395/) (2026)

## Abstract (arXiv)

Lexically constrained text generation aims to control the generated text by incorporating some pre-specified keywords into the output. Previous work injects lexical constraints into the output by controlling the decoding process or refining the candidate output iteratively, which tends to generate generic or ungrammatical sentences, and has high computational complexity. To address these challenges, we propose Constrained BART (CBART) for lexically constrained text generation. CBART leverages the pre-trained model BART and transfers part of the generation burden from the decoder to the encoder by decomposing this task into two sub-tasks, thereby improving the sentence quality. Concretely, we extend BART by adding a token-level classifier over the encoder, aiming at instructing the decoder where to replace and insert. Guided by the encoder, the decoder refines multiple tokens of the input in one step by inserting tokens before specific positions and re-predicting tokens with low confidence. To further reduce the inference latency, the decoder predicts all tokens in parallel. Experiment results on One-Billion-Word and Yelp show that CBART can generate plausible text with high quality and diversity while significantly accelerating inference.