# Toward Better Storylines with Sentence-Level Language Models

arXiv: https://arxiv.org/abs/2005.05255 · HTML: https://mlsanity.com/paper/2005.05255/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 14 of 14)

- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [Visualize Before You Write: Imagination-Guided Open-Ended Text Generation](https://mlsanity.com/paper/2210.03765/) (2022)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [ChapterBreak: A Challenge Dataset for Long-Range Language Models](https://mlsanity.com/paper/2204.10878/) (2022)
- [Learning to Plan for Language Modeling from Unlabeled Data](https://mlsanity.com/paper/2404.00614/) (2024)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [Are there identifiable structural parts in the sentence embedding whole?](https://mlsanity.com/paper/2406.16563/) (2024)
- [Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification](https://mlsanity.com/paper/2407.18119/) (2024)
- [rmC small IS2: A Simplified Commonsense Inference Evaluation for Story Prose](https://mlsanity.com/paper/2202.07880/) (2022)
- [Event Transition Planning for Open-ended Text Generation](https://mlsanity.com/paper/2204.09453/) (2022)
- [Controllable Fake Document Infilling for Cyber Deception](https://mlsanity.com/paper/2210.09917/) (2022)
- [InFillmore: Frame-Guided Language Generation with Bidirectional Context](https://mlsanity.com/paper/2103.04941/) (2021)
- [Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply](https://mlsanity.com/paper/2509.01606/) (2025)

## Abstract (arXiv)

We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. Rather than dealing with individual words, our method treats the story so far as a list of pre-trained sentence embeddings and predicts an embedding for the next sentence, which is more efficient than predicting word embeddings. Notably this allows us to consider a large number of candidates for the next sentence during training. We demonstrate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task and with promising results on larger-scale next sentence prediction tasks.