# A Novel Computational and Modeling Foundation for Automatic Coherence Assessment

arXiv: https://arxiv.org/abs/2310.00598 · HTML: https://mlsanity.com/paper/2310.00598/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 104 references that are in the map.
- First posted on arXiv 1 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [Learning to Extract Coherent Summary via Deep Reinforcement Learning](https://mlsanity.com/paper/1804.07036/) (2018)
- [A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks](https://mlsanity.com/paper/1611.01587/) (2016)
- [Trace Norm Regularised Deep Multi-Task Learning](https://mlsanity.com/paper/1606.04038/) (2016)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generation](https://mlsanity.com/paper/1808.06945/) (2018)
- [On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification](https://mlsanity.com/paper/2004.12617/) (2020)
- [Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators](https://mlsanity.com/paper/1904.13015/) (2019)
- [Neural Net Models for Open-Domain Discourse Coherence](https://mlsanity.com/paper/1606.01545/) (2016)
- [SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring](https://mlsanity.com/paper/1711.04981/) (2017)
- [Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations](https://mlsanity.com/paper/1909.00142/) (2019)
- [Deep Attentive Ranking Networks for Learning to Order Sentences](https://mlsanity.com/paper/2001.00056/) (2019)
- [Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://mlsanity.com/paper/2301.02724/) (2023)
- [Deep Enhanced Representation for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/1807.05154/) (2018)
- [Topological Sort for Sentence Ordering](https://mlsanity.com/paper/2005.00432/) (2020)
- [Discourse Coherence in the Wild: A Dataset, Evaluation and Methods](https://mlsanity.com/paper/1805.04993/) (2018)
- [A Cross-Domain Transferable Neural Coherence Model](https://mlsanity.com/paper/1905.11912/) (2019)
- [A Unified Neural Coherence Model](https://mlsanity.com/paper/1909.00349/) (2019)
- [Neural Sentence Ordering](https://mlsanity.com/paper/1607.06952/) (2016)
- [Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test](https://mlsanity.com/paper/2107.03448/) (2021)
- [Is Everything in Order? A Simple Way to Order Sentences](https://mlsanity.com/paper/2104.07064/) (2021)
- [Coherence Modeling of Asynchronous Conversations: A Neural Entity Grid Approach](https://mlsanity.com/paper/1805.02275/) (2018)
- [Extending Implicit Discourse Relation Recognition to the PDTB-3](https://mlsanity.com/paper/2010.06294/) (2020)
- [Text-based NP Enrichment](https://mlsanity.com/paper/2109.12085/) (2021)

## Built on by (most cited first, 4 of 4)

- [Attribute First, then Generate: Locally-attributable Grounded Text Generation](https://mlsanity.com/paper/2403.17104/) (2024)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [DEPTH: Discourse Education through Pre-Training Hierarchically](https://mlsanity.com/paper/2405.07788/) (2024)
- [BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation](https://mlsanity.com/paper/2405.17764/) (2024)

## Abstract (arXiv)

Coherence is an essential property of well-written texts, that refers to the way textual units relate to one another. In the era of generative AI, coherence assessment is essential for many NLP tasks; summarization, generation, long-form question-answering, and more. However, in NLP coherence is an ill-defined notion, not having a formal definition or evaluation metrics, that would allow for large-scale automatic and systematic coherence assessment. To bridge this gap, in this work we employ the formal linguistic definition of citetReinhart:1980 of what makes a discourse coherent, consisting of three conditions -- em cohesion, consistency and em relevance -- and formalize these conditions as respective computational tasks. We hypothesize that (i) a model trained on all of these tasks will learn the features required for coherence detection, and that (ii) a joint model for all tasks will exceed the performance of models trained on each task individually. On two benchmarks for coherence scoring rated by humans, one containing 500 automatically-generated short stories and another containing 4k real-world texts, our experiments confirm that jointly training on the proposed tasks leads to better performance on each task compared with task-specific models, and to better performance on assessing coherence overall, compared with strong baselines. We conclude that the formal and computational setup of coherence as proposed here provides a solid foundation for advanced methods of large-scale automatic assessment of coherence.