mlsanityData to 14 Oct 2026

Evaluating the Evaluators: Are readability metrics good measures of readability?

arXiv 2508.19221 ↗First posted 26 Aug 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Plain Language Summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. In this paper, we conduct a thorough survey of PLS literature, and identify that the current standard practice for readability evaluation is to use traditional readability metrics, such as Flesch-Kincaid Grade Level (FKGL). However, despite proven utility in other fields, these metrics have not been compared to human readability judgments in PLS. We evaluate 8 readability metrics and show that most correlate poorly with human judgments, including the most popular metric, FKGL. We then show that Language Models (LMs) are better judges of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. Extending our analysis to PLS datasets, which contain summaries aimed at non-expert audiences, we find that LMs better capture deeper measures of readability, such as required background knowledge, and lead to different conclusions than the traditional metrics. Based on these findings, we offer recommendations for best practices in the evaluation of plain language summaries. We release our analysis code and survey data.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 62 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

30 of 62 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  5. 2019Green AIcited by 627
  6. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  7. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  8. 2020CORD-19: The COVID-19 Open Research Datasetcited by 180
  9. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  10. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  11. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  12. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
18 more
  1. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  2. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  3. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  4. 2022Making Science Simple: Corpora for the Lay Summarisation of Scientific Literaturecited by 55
  5. 2020Automated Lay Language Summarization of Biomedical Scientific Reviewscited by 46
  6. 2019Plain English Summarization of Contractscited by 28
  7. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  8. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 24
  9. 2024RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Textscited by 23
  10. 2022Retrieval augmentation of large language models for lay language generationcited by 21
  11. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  12. 2024Large Language Models as Evaluators for Recommendation Explanationscited by 19
  13. 2024Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articlescited by 18
  14. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  15. 2023APPLS: Evaluating Evaluation Metrics for Plain Language Summarizationcited by 15
  16. 2024Know Your Audience: The benefits and pitfalls of generating plain language summaries beyond the "general" audiencecited by 13
  17. 2024SciNews: From Scholarly Complexities to Public Narratives -- A Dataset for Scientific News Report Generationcited by 12
  18. 2023Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decodingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Human-Agent Collaborative Paper-to-Page Craftingcited by 7
  2. 2026Disco-RAG: Discourse-Aware Retrieval-Augmented Generationcited by 1
  3. 2026Machine Learning Research Has Outpaced Its Communication Norms and NeurIPS Should Actcited by 0
  4. 2026Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responsescited by 0
  5. 2026Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generationcited by 0
  6. 2026Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Modelscited by 0