# Discourse Coherence in the Wild: A Dataset, Evaluation and Methods

arXiv: https://arxiv.org/abs/1805.04993 · HTML: https://mlsanity.com/paper/1805.04993/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2018.

## Built on by (most cited first, 20 of 20)

- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [Model Criticism for Long-Form Text Generation](https://mlsanity.com/paper/2210.08444/) (2022)
- [Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models](https://mlsanity.com/paper/2105.03495/) (2021)
- [Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation](https://mlsanity.com/paper/2104.05801/) (2021)
- [CoheSentia: A Novel Benchmark of Incremental versus Holistic Assessment of Coherence in Generated Texts](https://mlsanity.com/paper/2310.16329/) (2023)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [Evaluating Document Coherence Modelling](https://mlsanity.com/paper/2103.10133/) (2021)
- [RepEval: Effective Text Evaluation with LLM Representation](https://mlsanity.com/paper/2404.19563/) (2024)
- [SkillAggregation: Reference-free LLM-Dependent Aggregation](https://mlsanity.com/paper/2410.10215/) (2024)
- [Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks](https://mlsanity.com/paper/2306.06472/) (2023)
- [A Novel Computational and Modeling Foundation for Automatic Coherence Assessment](https://mlsanity.com/paper/2310.00598/) (2023)
- [BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence](https://mlsanity.com/paper/2312.16893/) (2023)
- [Transformer Models for Text Coherence Assessment](https://mlsanity.com/paper/2109.02176/) (2021)
- [DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting](https://mlsanity.com/paper/2406.19650/) (2024)
- [Joint Modeling of Entities and Discourse Relations for Coherence Assessment](https://mlsanity.com/paper/2509.04182/) (2025)
- [BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation](https://mlsanity.com/paper/2405.17764/) (2024)
- [Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking](https://mlsanity.com/paper/2507.08014/) (2025)
- [Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures](https://mlsanity.com/paper/2609.10893/) (2026)

## Abstract (arXiv)

To date there has been very little work on assessing discourse coherence methods on real-world data. To address this, we present a new corpus of real-world texts (GCDC) as well as the first large-scale evaluation of leading discourse coherence algorithms. We show that neural models, including two that we introduce here (SentAvg and ParSeq), tend to perform best. We analyze these performance differences and discuss patterns we observed in low coherence texts in four domains.