# Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting

arXiv: https://arxiv.org/abs/2609.24650 · HTML: https://mlsanity.com/paper/2609.24650/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Text Readability Assessment for Second Language Learners](https://mlsanity.com/paper/1906.07580/) (2019)
- [Supervised and Unsupervised Neural Approaches to Text Readability](https://mlsanity.com/paper/1907.11779/) (2019)
- [Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?](https://mlsanity.com/paper/2405.06105/) (2024)
- [Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations](https://mlsanity.com/paper/2508.03550/) (2025)
- [Measuring and Modifying the Readability of English Texts with GPT-4](https://mlsanity.com/paper/2410.14028/) (2024)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [Readability Reconsidered: A Cross-Dataset Analysis of Reference-Free Metrics](https://mlsanity.com/paper/2510.15345/) (2025)
- [Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods](https://mlsanity.com/paper/2502.11150/) (2025)

## Abstract (arXiv)

Readability assessment is essential for tailoring texts to intended audiences across educational, healthcare, and information retrieval domains. However, traditional readability formulas struggle to generalize across genres and languages, while supervised machine learning models rely on scarce, domain-specific annotated corpora, limiting their applicability--particularly for less-resourced languages. Large Language Models (LLMs) offer a highly scalable, multilingual alternative that requires no task-specific training, yet the impact of advanced prompting strategies on their performance remains underexplored. In this paper, we conduct a systematic benchmark of diverse open-source LLMs for multilingual readability assessment, focusing on the prediction of discrete readability levels required by educational frameworks. In addition to English, we evaluate our approach on a less-resourced language, Slovenian, to establish whether LLMs remain effective in low-resource settings. Specifically, we investigate the influence of explicit reasoning, demonstrating that Chain-of-Thought (CoT) prompting and reasoning-oriented models yield significant improvements over direct answering. Furthermore, our exploration of few-shot in-context learning reveals that providing just one labelled example per category (1-shot) substantially enhances prediction quality compared to zero-shot settings, with additional examples offering diminishing returns. By comprehensively comparing these approaches against traditional unsupervised metrics and state-of-the-art supervised baselines, we establish the viability of out-of-the-box LLMs as robust, cross-lingual readability assessors.