# Backward and Forward Language Modeling for Constrained Sentence Generation

arXiv: https://arxiv.org/abs/1512.06612 · HTML: https://mlsanity.com/paper/1512.06612/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2015.

## Built on by (most cited first, 9 of 9)

- [A Plug-and-Play Method for Controlled Text Generation](https://mlsanity.com/paper/2109.09707/) (2021)
- [Parallel Refinements for Lexically Constrained Text Generation with BART](https://mlsanity.com/paper/2109.12487/) (2021)
- [AmbiPun: Generating Humorous Puns with Ambiguous Context](https://mlsanity.com/paper/2205.01825/) (2022)
- [Arrows of Time for Large Language Models](https://mlsanity.com/paper/2401.17505/) (2024)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet](https://mlsanity.com/paper/2109.05797/) (2021)
- [Transformer-based Lexically Constrained Headline Generation](https://mlsanity.com/paper/2109.07080/) (2021)
- [A Survey of Pun Generation: Datasets, Evaluations and Methodologies](https://mlsanity.com/paper/2507.04793/) (2025)
- [Spiral Language Modeling](https://mlsanity.com/paper/2112.10543/) (2021)

## Abstract (arXiv)

Recent language models, especially those based on recurrent neural networks (RNNs), make it possible to generate natural language from a learned probability. Language generation has wide applications including machine translation, summarization, question answering, conversation systems, etc. Existing methods typically learn a joint probability of words conditioned on additional information, which is (either statically or dynamically) fed to RNN's hidden layer. In many applications, we are likely to impose hard constraints on the generated texts, i.e., a particular word must appear in the sentence. Unfortunately, existing approaches could not solve this problem. In this paper, we propose a novel backward and forward language model. Provided a specific word, we use RNNs to generate previous words and future words, either simultaneously or asynchronously, resulting in two model variants. In this way, the given word could appear at any position in the sentence. Experimental results show that the generated texts are comparable to sequential LMs in quality.