# Improving Language Generation with Sentence Coherence Objective

arXiv: https://arxiv.org/abs/2009.06358 · HTML: https://mlsanity.com/paper/2009.06358/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning](https://mlsanity.com/paper/2212.10341/) (2022)
- [Know When To Stop: A Study of Semantic Drift in Text Generation](https://mlsanity.com/paper/2404.05411/) (2024)
- [Unsupervised Extractive Opinion Summarization Using Sparse Coding](https://mlsanity.com/paper/2203.07921/) (2022)
- [Recent Trends in Unsupervised Summarization](https://mlsanity.com/paper/2305.11231/) (2023)
- [Evaluating the Capabilities of LLMs for Supporting Anticipatory Impact Assessment](https://mlsanity.com/paper/2401.18028/) (2024)

## Abstract (arXiv)

Conditional story generation and contextual text continuation have become increasingly popular topics in NLP community. Existing models are often prone to output paragraphs of texts that gradually diverge from the given prompt. Although the generated text may have a reasonable perplexity and diversity, it could easily be identified by human as gibberish. The goal of our project is to improve the coherence and consistency across sentences in a language-generation model. We aim to solve this issue by first training a sentence pair coherence classifier with GPT-2 pretrained model, and then co-train the GPT-2 language model with this new coherence objective using a method analogous to the REINFORCE algorithm. This fine-tuned language model is able to generate lengthy paragraph conditioned on a given topic without diverging too much. The simplicity of this model allows it to be applicable to a variety of underlying language model architecture since it only modifies the final layer of the pre-trained model.