# Evaluating Text Coherence at Sentence and Paragraph Levels

arXiv: https://arxiv.org/abs/2006.03221 · HTML: https://mlsanity.com/paper/2006.03221/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2020.

## Built on by (most cited first, 7 of 7)

- [SIRE: Separate Intra- and Inter-sentential Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2106.01709/) (2021)
- [BERT4SO: Neural Sentence Ordering by Fine-tuning BERT](https://mlsanity.com/paper/2103.13584/) (2021)
- [GisPy: A Tool for Measuring Gist Inference Score in Text](https://mlsanity.com/paper/2205.12484/) (2022)
- [LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability](https://mlsanity.com/paper/2510.24345/) (2025)
- [Local and Global Context-Based Pairwise Models for Sentence Ordering](https://mlsanity.com/paper/2110.04291/) (2021)
- [CoUDA: Coherence Evaluation via Unified Data Augmentation](https://mlsanity.com/paper/2404.00681/) (2024)
- [CAGE: Coherence-Aware Graph Encoding for Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.04647/) (2026)

## Abstract (arXiv)

In this paper, to evaluate text coherence, we propose the paragraph ordering task as well as conducting sentence ordering. We collected four distinct corpora from different domains on which we investigate the adaptation of existing sentence ordering methods to a paragraph ordering task. We also compare the learnability and robustness of existing models by artificially creating mini datasets and noisy datasets respectively and verifying the efficiency of established models under these circumstances. Furthermore, we carry out human evaluation on the rearranged passages from two competitive models and confirm that WLCS-l is a better metric performing significantly higher correlations with human rating than tau, the most prevalent metric used before. Results from these evaluations show that except for certain extreme conditions, the recurrent graph neural network-based model is an optimal choice for coherence modeling.