# Local and Global Context-Based Pairwise Models for Sentence Ordering

arXiv: https://arxiv.org/abs/2110.04291 · HTML: https://mlsanity.com/paper/2110.04291/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 54 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://mlsanity.com/paper/1804.09541/) (2018)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [What Makes A Good Story? Designing Composite Rewards for Visual Storytelling](https://mlsanity.com/paper/1909.05316/) (2019)
- [SLM: Learning a Discourse Language Representation with Sentence Unshuffling](https://mlsanity.com/paper/2010.16249/) (2020)
- [Neural Net Models for Open-Domain Discourse Coherence](https://mlsanity.com/paper/1606.01545/) (2016)
- [Deep Attentive Ranking Networks for Learning to Order Sentences](https://mlsanity.com/paper/2001.00056/) (2019)
- [Topological Sort for Sentence Ordering](https://mlsanity.com/paper/2005.00432/) (2020)
- [Graph-based Neural Sentence Ordering](https://mlsanity.com/paper/1912.07225/) (2019)
- [Neural Sentence Ordering](https://mlsanity.com/paper/1607.06952/) (2016)
- [KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning](https://mlsanity.com/paper/2012.07000/) (2020)
- [End-to-End Neural Sentence Ordering Using Pointer Network](https://mlsanity.com/paper/1611.04953/) (2016)
- [Sort Story: Sorting Jumbled Images and Captions into Stories](https://mlsanity.com/paper/1606.07493/) (2016)
- [Towards Content Transfer through Grounded Text Generation](https://mlsanity.com/paper/1905.05293/) (2019)
- [Neural Sentence Ordering Based on Constraint Graphs](https://mlsanity.com/paper/2101.11178/) (2021)
- [Evaluating Text Coherence at Sentence and Paragraph Levels](https://mlsanity.com/paper/2006.03221/) (2020)
- [BERT4SO: Neural Sentence Ordering by Fine-tuning BERT](https://mlsanity.com/paper/2103.13584/) (2021)

## Built on by (most cited first, 1 of 1)

- [Associative Learning Mechanism for Drug-Target Interaction Prediction](https://mlsanity.com/paper/2205.15364/) (2022)

## Abstract (arXiv)

Sentence Ordering refers to the task of rearranging a set of sentences into the appropriate coherent order. For this task, most previous approaches have explored global context-based end-to-end methods using Sequence Generation techniques. In this paper, we put forward a set of robust local and global context-based pairwise ordering strategies, leveraging which our prediction strategies outperform all previous works in this domain. Our proposed encoding method utilizes the paragraph's rich global contextual information to predict the pairwise order using novel transformer architectures. Analysis of the two proposed decoding strategies helps better explain error propagation in pairwise models. This approach is the most accurate pure pairwise model and our encoding strategy also significantly improves the performance of other recent approaches that use pairwise models, including the previous state-of-the-art, demonstrating the research novelty and generalizability of this work. Additionally, we show how the pre-training task for ALBERT helps it to significantly outperform BERT, despite having considerably lesser parameters. The extensive experimental results, architectural analysis and ablation studies demonstrate the effectiveness and superiority of the proposed models compared to the previous state-of-the-art, besides providing a much better understanding of the functioning of pairwise models.