# MedReadMe: A Systematic Study for Fine-grained Sentence Readability in Medical Domain

arXiv: https://arxiv.org/abs/2405.02144 · HTML: https://mlsanity.com/paper/2405.02144/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2024.

## Built on by (most cited first, 7 of 7)

- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [ReadCtrl: Personalizing text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2406.09205/) (2024)
- [LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction](https://mlsanity.com/paper/2408.12249/) (2024)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA](https://mlsanity.com/paper/2604.05051/) (2026)
- [The Crutch or the Ceiling? How Different Generations of LLMs Shape EFL Student Writings](https://mlsanity.com/paper/2604.15460/) (2026)

## Abstract (arXiv)

Medical texts are notoriously challenging to read. Properly measuring their readability is the first step towards making them more accessible. In this paper, we present a systematic study on fine-grained readability measurements in the medical domain at both sentence-level and span-level. We introduce a new dataset MedReadMe, which consists of manually annotated readability ratings and fine-grained complex span annotation for 4,520 sentences, featuring two novel "Google-Easy" and "Google-Hard" categories. It supports our quantitative analysis, which covers 650 linguistic features and automatic complex word and jargon identification. Enabled by our high-quality annotation, we benchmark and improve several state-of-the-art sentence-level readability metrics for the medical domain specifically, which include unsupervised, supervised, and prompting-based methods using recently developed large language models (LLMs). Informed by our fine-grained complex span annotation, we find that adding a single feature, capturing the number of jargon spans, into existing readability formulas can significantly improve their correlation with human judgments. The data is available at tinyurl.com/medreadme-repo