# Measuring and Modifying the Readability of English Texts with GPT-4

arXiv: https://arxiv.org/abs/2410.14028 · HTML: https://mlsanity.com/paper/2410.14028/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization](https://mlsanity.com/paper/2311.09184/) (2023)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [Supervised and Unsupervised Neural Approaches to Text Readability](https://mlsanity.com/paper/1907.11779/) (2019)
- [Linguistic Features for Readability Assessment](https://mlsanity.com/paper/2006.00377/) (2020)
- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [Trends, Limitations and Open Challenges in Automatic Readability Assessment Research](https://mlsanity.com/paper/2105.00973/) (2021)
- [Generating Summaries with Controllable Readability Levels](https://mlsanity.com/paper/2310.10623/) (2023)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [A task in a suit and a tie: paraphrase generation with semantic augmentation](https://mlsanity.com/paper/1811.00119/) (2018)
- [(Psycho-)Linguistic Features Meet Transformer Models for Improved Explainable and Controllable Text Simplification](https://mlsanity.com/paper/2212.09848/) (2022)

## Built on by (most cited first, 10 of 10)

- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanations](https://mlsanity.com/paper/2507.05179/) (2025)
- [Rethinking the Role of Text Complexity in Language Model Pretraining](https://mlsanity.com/paper/2509.16551/) (2025)
- [Readability Reconsidered: A Cross-Dataset Analysis of Reference-Free Metrics](https://mlsanity.com/paper/2510.15345/) (2025)
- [Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods](https://mlsanity.com/paper/2502.11150/) (2025)
- [Linguistic and Argument Diversity in Synthetic Data for Function-Calling Agents](https://mlsanity.com/paper/2601.17829/) (2026)
- [DMAP: A Distribution Map for Text](https://mlsanity.com/paper/2602.11871/) (2026)
- [Can LLMs Control Readability? A Multi-Dimensional Evaluation Framework for CEFR-Controlled Arabic Generation](https://mlsanity.com/paper/2606.21981/) (2026)
- [Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting](https://mlsanity.com/paper/2609.24650/) (2026)

## Abstract (arXiv)

The success of Large Language Models (LLMs) in other domains has raised the question of whether LLMs can reliably assess and manipulate the readability of text. We approach this question empirically. First, using a published corpus of 4,724 English text excerpts, we find that readability estimates produced ``zero-shot'' from GPT-4 Turbo and GPT-4o mini exhibit relatively high correlation with human judgments (r = 0.76 and r = 0.74, respectively), out-performing estimates derived from traditional readability formulas and various psycholinguistic indices. Then, in a pre-registered human experiment (N = 59), we ask whether Turbo can reliably make text easier or harder to read. We find evidence to support this hypothesis, though considerable variance in human judgments remains unexplained. We conclude by discussing the limitations of this approach, including limited scope, as well as the validity of the ``readability'' construct and its dependence on context, audience, and goal.