# Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test

arXiv: https://arxiv.org/abs/2107.03448 · HTML: https://mlsanity.com/paper/2107.03448/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [Abstractive Summarization of Reddit Posts with Multi-level Memory Networks](https://mlsanity.com/paper/1811.00783/) (2018)
- [A Unified Neural Coherence Model](https://mlsanity.com/paper/1909.00349/) (2019)
- [Coherence Modeling of Asynchronous Conversations: A Neural Entity Grid Approach](https://mlsanity.com/paper/1805.02275/) (2018)
- [Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks](https://mlsanity.com/paper/2004.14626/) (2020)

## Built on by (most cited first, 17 of 17)

- [HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](https://mlsanity.com/paper/2311.09528/) (2023)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models](https://mlsanity.com/paper/2305.03573/) (2023)
- [Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence](https://mlsanity.com/paper/2402.10175/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Combat AI With AI: Counteract Machine-Generated Fake Restaurant Reviews on Social Media](https://mlsanity.com/paper/2302.07731/) (2023)
- [How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation](https://mlsanity.com/paper/2209.06517/) (2022)
- [A Novel Computational and Modeling Foundation for Automatic Coherence Assessment](https://mlsanity.com/paper/2310.00598/) (2023)
- [A Digital Language Coherence Marker for Monitoring Dementia](https://mlsanity.com/paper/2310.09623/) (2023)
- [BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence](https://mlsanity.com/paper/2312.16893/) (2023)
- [GisPy: A Tool for Measuring Gist Inference Score in Text](https://mlsanity.com/paper/2205.12484/) (2022)
- [Joint Modeling of Entities and Discourse Relations for Coherence Assessment](https://mlsanity.com/paper/2509.04182/) (2025)
- [Imagined versus Remembered Stories: Quantifying Differences in Narrative Flow](https://mlsanity.com/paper/2201.02662/) (2022)
- [An Application of Pseudo-Log-Likelihoods to Natural Language Scoring](https://mlsanity.com/paper/2201.09377/) (2022)
- [Enhancing Coherence of Extractive Summarization with Multitask Learning](https://mlsanity.com/paper/2305.12851/) (2023)
- [Disambiguation of Emotion Annotations by Contextualizing Events in Plausible Narratives](https://mlsanity.com/paper/2508.09954/) (2025)
- [On Non-interactive Evaluation of Animal Communication Translators](https://mlsanity.com/paper/2510.15768/) (2025)

## Abstract (arXiv)

The Shuffle Test is the most common task to evaluate whether NLP models can measure coherence in text. Most recent work uses direct supervision on the task; we show that by simply finetuning a RoBERTa model, we can achieve a near perfect accuracy of 97.8%, a state-of-the-art. We argue that this outstanding performance is unlikely to lead to a good model of text coherence, and suggest that the Shuffle Test should be approached in a Zero-Shot setting: models should be evaluated without being trained on the task itself. We evaluate common models in this setting, such as Generative and Bi-directional Transformers, and find that larger architectures achieve high-performance out-of-the-box. Finally, we suggest the k-Block Shuffle Test, a modification of the original by increasing the size of blocks shuffled. Even though human reader performance remains high (around 95% accuracy), model performance drops from 94% to 78% as block size increases, creating a conceptually simple challenge to benchmark NLP models. Code available: https://github.com/tingofurro/shuffle_test/