# Readability Reconsidered: A Cross-Dataset Analysis of Reference-Free Metrics

arXiv: https://arxiv.org/abs/2510.15345 · HTML: https://mlsanity.com/paper/2510.15345/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice](https://mlsanity.com/paper/2402.01864/) (2024)
- [Text Readability Assessment for Second Language Learners](https://mlsanity.com/paper/1906.07580/) (2019)
- [Evaluating Scoring Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2506.22316/) (2025)
- [CEFR-Based Sentence Difficulty Annotation and Assessment](https://mlsanity.com/paper/2210.11766/) (2022)
- [Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure](https://mlsanity.com/paper/2410.03781/) (2024)
- [Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models](https://mlsanity.com/paper/2504.14194/) (2025)
- [ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment](https://mlsanity.com/paper/2305.14463/) (2023)
- [Know Your Audience: The benefits and pitfalls of generating plain language summaries beyond the "general" audience](https://mlsanity.com/paper/2403.04979/) (2024)
- [Measuring and Modifying the Readability of English Texts with GPT-4](https://mlsanity.com/paper/2410.14028/) (2024)
- [Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts](https://mlsanity.com/paper/2405.09482/) (2024)
- [ELI-Why: Evaluating the Pedagogical Utility of Language Model Explanations](https://mlsanity.com/paper/2506.14200/) (2025)

## Built on by (most cited first, 2 of 2)

- [Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models](https://mlsanity.com/paper/2608.23327/) (2026)
- [Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting](https://mlsanity.com/paper/2609.24650/) (2026)

## Abstract (arXiv)

Automatic readability assessment plays a key role in ensuring effective and accessible written communication. Despite significant progress, the field is hindered by inconsistent definitions of readability and measurements that rely on surface-level text properties. In this work, we investigate the factors shaping human perceptions of readability through the analysis of 897 judgments, finding that, beyond surface-level cues, information content and topic strongly shape text comprehensibility. Furthermore, we evaluate 15 popular readability metrics across five English datasets, contrasting them with six more nuanced, model-based metrics. Our results show that four model-based metrics consistently place among the top four in rank correlations with human judgments, while the best performing traditional metric achieves an average rank of 8.6. These findings highlight a mismatch between current readability metrics and human perceptions, pointing to model-based approaches as a more promising direction.