# Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models

arXiv: https://arxiv.org/abs/2309.05454 · HTML: https://mlsanity.com/paper/2309.05454/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 72 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 30 of 39)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [Text Readability Assessment for Second Language Learners](https://mlsanity.com/paper/1906.07580/) (2019)

## Built on by (most cited first, 17 of 17)

- [From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation](https://mlsanity.com/paper/2406.03030/) (2024)
- [Analysing Zero-Shot Readability-Controlled Sentence Simplification](https://mlsanity.com/paper/2409.20246/) (2024)
- [Measuring and Modifying the Readability of English Texts with GPT-4](https://mlsanity.com/paper/2410.14028/) (2024)
- [Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts](https://mlsanity.com/paper/2405.09482/) (2024)
- [Standardize: Aligning Language Models with Expert-Defined Standards for Content Generation](https://mlsanity.com/paper/2402.12593/) (2024)
- [Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring](https://mlsanity.com/paper/2505.08351/) (2025)
- [Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance](https://mlsanity.com/paper/2503.04736/) (2025)
- [Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversation](https://mlsanity.com/paper/2506.04072/) (2025)
- [Controlling Reading Ease with Gaze-Guided Text Generation](https://mlsanity.com/paper/2601.17781/) (2026)
- [SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning](https://mlsanity.com/paper/2407.13297/) (2024)
- [Rethinking the Role of Text Complexity in Language Model Pretraining](https://mlsanity.com/paper/2509.16551/) (2025)
- [Classroom AI: Large Language Models as Grade-Specific Teachers](https://mlsanity.com/paper/2601.06225/) (2026)
- [Let's Simplify Step by Step: Guiding LLM Towards Multilingual Unsupervised Proficiency-Controlled Sentence Simplification](https://mlsanity.com/paper/2602.07499/) (2026)
- [OUNLP at TSAR 2025 Shared Task: Multi-Round Text Simplifier via Code Generation](https://mlsanity.com/paper/2511.04495/) (2025)
- [Readability Measures and Automatic Text Simplification: In the Search of a Construct](https://mlsanity.com/paper/2511.09536/) (2025)
- [Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responses](https://mlsanity.com/paper/2606.06788/) (2026)
- [Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models](https://mlsanity.com/paper/2608.23327/) (2026)

## Abstract (arXiv)

Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives--tasks that teachers perform--using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5--which have shown promising results.