# Using BERT Encoding and Sentence-Level Language Model for Sentence Ordering

arXiv: https://arxiv.org/abs/2108.10986 · HTML: https://mlsanity.com/paper/2108.10986/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 24 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Discourse-Aware Neural Rewards for Coherent Text Generation](https://mlsanity.com/paper/1805.03766/) (2018)
- [Neural Net Models for Open-Domain Discourse Coherence](https://mlsanity.com/paper/1606.01545/) (2016)
- [Deep Attentive Ranking Networks for Learning to Order Sentences](https://mlsanity.com/paper/2001.00056/) (2019)
- [End-to-End Neural Sentence Ordering Using Pointer Network](https://mlsanity.com/paper/1611.04953/) (2016)
- [Sort Story: Sorting Jumbled Images and Captions into Stories](https://mlsanity.com/paper/1606.07493/) (2016)
- [Using Sentence-Level LSTM Language Models for Script Inference](https://mlsanity.com/paper/1604.02993/) (2016)

## Built on by (most cited first, 4 of 4)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Hallucination-minimized Data-to-answer Framework for Financial Decision-makers](https://mlsanity.com/paper/2311.07592/) (2023)
- [Pruned Graph Neural Network for Short Story Ordering](https://mlsanity.com/paper/2203.06778/) (2022)

## Abstract (arXiv)

Discovering the logical sequence of events is one of the cornerstones in Natural Language Understanding. One approach to learn the sequence of events is to study the order of sentences in a coherent text. Sentence ordering can be applied in various tasks such as retrieval-based Question Answering, document summarization, storytelling, text generation, and dialogue systems. Furthermore, we can learn to model text coherence by learning how to order a set of shuffled sentences. Previous research has relied on RNN, LSTM, and BiLSTM architecture for learning text language models. However, these networks have performed poorly due to the lack of attention mechanisms. We propose an algorithm for sentence ordering in a corpus of short stories. Our proposed method uses a language model based on Universal Transformers (UT) that captures sentences' dependencies by employing an attention mechanism. Our method improves the previous state-of-the-art in terms of Perfect Match Ratio (PMR) score in the ROCStories dataset, a corpus of nearly 100K short human-made stories. The proposed model includes three components: Sentence Encoder, Language Model, and Sentence Arrangement with Brute Force Search. The first component generates sentence embeddings using SBERT-WK pre-trained model fine-tuned on the ROCStories data. Then a Universal Transformer network generates a sentence-level language model. For decoding, the network generates a candidate sentence as the following sentence of the current sentence. We use cosine similarity as a scoring function to assign scores to the candidate embedding and the embeddings of other sentences in the shuffled set. Then a Brute Force Search is employed to maximize the sum of similarities between pairs of consecutive sentences.