# Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet arXiv: https://arxiv.org/abs/2109.05797 ยท HTML: https://mlsanity.com/paper/2109.05797/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 38 references that are in the map. - First posted on arXiv 13 Sep 2021. ## Builds on (most cited first, 24 of 24) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018) - [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019) - [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018) - [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019) - [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017) - [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019) - [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018) - [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018) - [CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling](https://mlsanity.com/paper/1811.10996/) (2018) - [Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation](https://mlsanity.com/paper/1607.00970/) (2016) - [Text Infilling](https://mlsanity.com/paper/1901.00158/) (2019) - [Image Captioning with Deep Bidirectional LSTMs](https://mlsanity.com/paper/1604.00790/) (2016) - [Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series](https://mlsanity.com/paper/1504.01575/) (2015) - [TIGS: An Inference Algorithm for Text Infilling with Gradient Search](https://mlsanity.com/paper/1905.10752/) (2019) - [Backward and Forward Language Modeling for Constrained Sentence Generation](https://mlsanity.com/paper/1512.06612/) (2015) - [BFGAN: Backward and Forward Generative Adversarial Networks for Lexically Constrained Sentence Generation](https://mlsanity.com/paper/1806.08097/) (2018) - [Incorporating Discriminator in Sentence Generation: a Gibbs Sampling Method](https://mlsanity.com/paper/1802.08970/) (2018) ## Built on by (most cited first, 10 of 10) - [Parallel Refinements for Lexically Constrained Text Generation with BART](https://mlsanity.com/paper/2109.12487/) (2021) - [ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions](https://mlsanity.com/paper/2511.14342/) (2025) - [Unsupervised Editing for Counterfactual Stories](https://mlsanity.com/paper/2112.05417/) (2021) - [Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning](https://mlsanity.com/paper/2210.11708/) (2022) - [Improving Factual Error Correction by Learning to Inject Factual Errors](https://mlsanity.com/paper/2312.07049/) (2023) - [Lexical Complexity Controlled Sentence Generation](https://mlsanity.com/paper/2211.14540/) (2022) - [Pseudo-labeling with Keyword Refining for Few-Supervised Video Captioning](https://mlsanity.com/paper/2411.04059/) (2024) - [Evaluating Human-LLM Representation Alignment: A Case Study on Affective Sentence Generation for Augmentative and Alternative Communication](https://mlsanity.com/paper/2503.11881/) (2025) - [A Benchmark for Text Expansion: Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2309.09198/) (2023) - [Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models](https://mlsanity.com/paper/2310.16343/) (2023) ## Abstract (arXiv) Lexically constrained sentence generation allows the incorporation of prior knowledge such as lexical constraints into the output. This technique has been applied to machine translation, and dialog response generation. Previous work usually used Markov Chain Monte Carlo (MCMC) sampling to generate lexically constrained sentences, but they randomly determined the position to be edited and the action to be taken, resulting in many invalid refinements. To overcome this challenge, we used a classifier to instruct the MCMC-based models where and how to refine the candidate sentences. First, we developed two methods to create synthetic data on which the pre-trained model is fine-tuned to obtain a reliable classifier. Next, we proposed a two-step approach, "Predict and Revise", for constrained sentence generation. During the predict step, we leveraged the classifier to compute the learned prior for the candidate sentence. During the revise step, we resorted to MCMC sampling to revise the candidate sentence by conducting a sampled action at a sampled position drawn from the learned prior. We compared our proposed models with many strong baselines on two tasks, generating sentences with lexical constraints and text infilling. Experimental results have demonstrated that our proposed model performs much better than the previous work in terms of sentence fluency and diversity. Our code and pre-trained models are available at https://github.com/NLPCode/MCMCXLNet.