# Transformer Models for Text Coherence Assessment

arXiv: https://arxiv.org/abs/2109.02176 · HTML: https://mlsanity.com/paper/2109.02176/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 5 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Learning to Write with Cooperative Discriminators](https://mlsanity.com/paper/1805.06087/) (2018)
- [Neural Net Models for Open-Domain Discourse Coherence](https://mlsanity.com/paper/1606.01545/) (2016)
- [Discourse Coherence in the Wild: A Dataset, Evaluation and Methods](https://mlsanity.com/paper/1805.04993/) (2018)
- [A Cross-Domain Transferable Neural Coherence Model](https://mlsanity.com/paper/1905.11912/) (2019)
- [A Unified Neural Coherence Model](https://mlsanity.com/paper/1909.00349/) (2019)
- [Neural Automated Essay Scoring and Coherence Modeling for Adversarially Crafted Input](https://mlsanity.com/paper/1804.06898/) (2018)
- [Modeling Coherence for Discourse Neural Machine Translation](https://mlsanity.com/paper/1811.05683/) (2018)
- [Coherence Modeling of Asynchronous Conversations: A Neural Entity Grid Approach](https://mlsanity.com/paper/1805.02275/) (2018)
- [Multi-Task Learning for Coherence Modeling](https://mlsanity.com/paper/1907.02427/) (2019)
- [Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks](https://mlsanity.com/paper/2004.14626/) (2020)
- [Neural RST-based Evaluation of Discourse Coherence](https://mlsanity.com/paper/2009.14463/) (2020)
- [Analyzing Neural Discourse Coherence Models](https://mlsanity.com/paper/2011.06306/) (2020)

## Built on by (most cited first, 3 of 3)

- [Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting](https://mlsanity.com/paper/2305.14755/) (2023)
- [Joint Modeling of Entities and Discourse Relations for Coherence Assessment](https://mlsanity.com/paper/2509.04182/) (2025)
- [Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures](https://mlsanity.com/paper/2609.10893/) (2026)

## Abstract (arXiv)

Coherence is an important aspect of text quality and is crucial for ensuring its readability. It is essential desirable for outputs from text generation systems like summarization, question answering, machine translation, question generation, table-to-text, etc. An automated coherence scoring model is also helpful in essay scoring or providing writing feedback. A large body of previous work has leveraged entity-based methods, syntactic patterns, discourse relations, and more recently traditional deep learning architectures for text coherence assessment. Previous work suffers from drawbacks like the inability to handle long-range dependencies, out-of-vocabulary words, or model sequence information. We hypothesize that coherence assessment is a cognitively complex task that requires deeper models and can benefit from other related tasks. Accordingly, in this paper, we propose four different Transformer-based architectures for the task: vanilla Transformer, hierarchical Transformer, multi-task learning-based model, and a model with fact-based input representation. Our experiments with popular benchmark datasets across multiple domains on four different coherence assessment tasks demonstrate that our models achieve state-of-the-art results outperforming existing models by a good margin.