mlsanityData to 14 Oct 2026

Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models

arXiv 2309.05454 ↗First posted 11 Sep 2023cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives--tasks that teachers perform--using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5--which have shown promising results.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

39 of 72 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
27 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  7. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  8. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  9. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  10. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  11. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  13. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  14. 2020A Distributional Approach to Controlled Text Generationcited by 119
  15. 2022The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialoguescited by 79
  16. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  17. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  18. 2019Text Readability Assessment for Second Language Learnerscited by 54
  19. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  20. 2020Controllable Text Simplification with Explicit Paraphrasingcited by 53
  21. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 50
  22. 2023Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instructioncited by 37
  23. 2019An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answeringcited by 22
  24. 2020Decoding Methods for Neural Narrative Generationcited by 18
  25. 2023Teaching the Pre-trained Model to Generate Simple Texts for Text Simplificationcited by 10
  26. 2021Changing the Mind of Transformers for Topically-Controllable Language Generationcited by 7
  27. 2022Uniform Complexity for Text Generationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 5.

17 papers.

Sort:

By area:

  1. 2024From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generationcited by 15
  2. 2024Analysing Zero-Shot Readability-Controlled Sentence Simplificationcited by 12
  3. 2024Measuring and Modifying the Readability of English Texts with GPT-4cited by 10
  4. 2024Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Textscited by 9
  5. 2024Standardize: Aligning Language Models with Expert-Defined Standards for Content Generationcited by 7
  6. 2025Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoringcited by 7
  7. 2025Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliancecited by 5
  8. 2025Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversationcited by 3
  9. 2026Controlling Reading Ease with Gaze-Guided Text Generationcited by 3
  10. 2024SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learningcited by 2
  11. 2025Rethinking the Role of Text Complexity in Language Model Pretrainingcited by 2
  12. 2026Classroom AI: Large Language Models as Grade-Specific Teacherscited by 2
5 more
  1. 2026Let's Simplify Step by Step: Guiding LLM Towards Multilingual Unsupervised Proficiency-Controlled Sentence Simplificationcited by 1
  2. 2025OUNLP at TSAR 2025 Shared Task: Multi-Round Text Simplifier via Code Generationcited by 0
  3. 2025Readability Measures and Automatic Text Simplification: In the Search of a Constructcited by 0
  4. 2026Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responsescited by 0
  5. 2026Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting
  2. 2025A Course Correction in Steerability Evaluation: Revealing Miscalibration and Side Effects in LLMs
  3. 2025Readability ne Learnability: Rethinking the Role of Simplicity in Training Small Language Models
  4. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  5. 2023CITING: Large Language Models Create Curriculum for Instruction Tuning
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
  7. 2023Assessing the efficacy of large language models in generating accurate teacher responses
  8. 2025Human-Aligned Code Readability Assessment with Large Language Models ↗