# BERT4SO: Neural Sentence Ordering by Fine-tuning BERT

arXiv: https://arxiv.org/abs/2103.13584 · HTML: https://mlsanity.com/paper/2103.13584/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://mlsanity.com/paper/1907.04829/) (2019)
- [Deep Attentive Ranking Networks for Learning to Order Sentences](https://mlsanity.com/paper/2001.00056/) (2019)
- [Topological Sort for Sentence Ordering](https://mlsanity.com/paper/2005.00432/) (2020)
- [Graph-based Neural Sentence Ordering](https://mlsanity.com/paper/1912.07225/) (2019)
- [Neural Sentence Ordering](https://mlsanity.com/paper/1607.06952/) (2016)
- [End-to-End Neural Sentence Ordering Using Pointer Network](https://mlsanity.com/paper/1611.04953/) (2016)
- [ScriptWriter: Narrative-Guided Script Generation](https://mlsanity.com/paper/2005.10331/) (2020)
- [Neural Sentence Ordering Based on Constraint Graphs](https://mlsanity.com/paper/2101.11178/) (2021)
- [Deep Ranking Model by Large Adaptive Margin Learning for Person Re-identification](https://mlsanity.com/paper/1707.00409/) (2017)
- [Evaluating Text Coherence at Sentence and Paragraph Levels](https://mlsanity.com/paper/2006.03221/) (2020)

## Built on by (most cited first, 2 of 2)

- [Kanbun-LM: Reading and Translating Classical Chinese in Japanese Methods by Language Models](https://mlsanity.com/paper/2305.12759/) (2023)
- [Local and Global Context-Based Pairwise Models for Sentence Ordering](https://mlsanity.com/paper/2110.04291/) (2021)

## Abstract (arXiv)

Sentence ordering aims to arrange the sentences of a given text in the correct order. Recent work frames it as a ranking problem and applies deep neural networks to it. In this work, we propose a new method, named BERT4SO, by fine-tuning BERT for sentence ordering. We concatenate all sentences and compute their representations by using multiple special tokens and carefully designed segment (interval) embeddings. The tokens across multiple sentences can attend to each other which greatly enhances their interactions. We also propose a margin-based listwise ranking loss based on ListMLE to facilitate the optimization process. Experimental results on five benchmark datasets demonstrate the effectiveness of our proposed method.