# Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension

arXiv: https://arxiv.org/abs/2312.10126 · HTML: https://mlsanity.com/paper/2312.10126/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 82 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [BLEU is Not Suitable for the Evaluation of Text Simplification](https://mlsanity.com/paper/1810.05995/) (2018)
- [Controllable Sentence Simplification](https://mlsanity.com/paper/1910.02677/) (2019)
- [What Makes Reading Comprehension Questions Easier?](https://mlsanity.com/paper/1808.09384/) (2018)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://mlsanity.com/paper/2212.09739/) (2022)
- [Paragraph-level Simplification of Medical Texts](https://mlsanity.com/paper/2104.05767/) (2021)
- [Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text](https://mlsanity.com/paper/2107.03444/) (2021)
- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [Evaluating Factuality in Text Simplification](https://mlsanity.com/paper/2204.07562/) (2022)
- [An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models](https://mlsanity.com/paper/2203.09486/) (2022)
- [STARC: Structured Annotations for Reading Comprehension](https://mlsanity.com/paper/2004.14797/) (2020)
- [Shortcomings of Question Answering Based Factuality Frameworks for Error Localization](https://mlsanity.com/paper/2210.06748/) (2022)
- [Sentence Simplification via Large Language Models](https://mlsanity.com/paper/2302.11957/) (2023)
- [How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks](https://mlsanity.com/paper/2109.04604/) (2021)
- [Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation](https://mlsanity.com/paper/2005.11197/) (2020)
- [Predicting Text Readability from Scrolling Interactions](https://mlsanity.com/paper/2105.06354/) (2021)

## Built on by (most cited first, 16 of 16)

- [InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification](https://mlsanity.com/paper/2401.16475/) (2024)
- [Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering](https://mlsanity.com/paper/2504.07583/) (2025)
- [LLMs in Education: Novel Perspectives, Challenges, and Opportunities](https://mlsanity.com/paper/2409.11917/) (2024)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Steering Large Language Models with Register Analysis for Arbitrary Style Transfer](https://mlsanity.com/paper/2505.00679/) (2025)
- [LLM-based Text Simplification and its Effect on User Comprehension and Cognitive Load](https://mlsanity.com/paper/2505.01980/) (2025)
- [It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledge](https://mlsanity.com/paper/2509.16107/) (2025)
- [Simplifications are Absolutists: How Simplified Language Reduces Word Sense Awareness in LLM-Generated Definitions](https://mlsanity.com/paper/2507.11981/) (2025)
- [Towards Trustworthy Lexical Simplification: Exploring Safety and Efficiency with Small LLMs](https://mlsanity.com/paper/2509.25086/) (2025)
- [BiST: A Gold Standard Bangla-English Bilingual Corpus for Sentence Structure and Tense Classification with Inter-Annotator Agreement](https://mlsanity.com/paper/2604.04708/) (2026)
- [Evaluating the Effectiveness of Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities](https://mlsanity.com/paper/2507.01479/) (2025)
- [JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences](https://mlsanity.com/paper/2508.16870/) (2025)
- [Readability Measures and Automatic Text Simplification: In the Search of a Construct](https://mlsanity.com/paper/2511.09536/) (2025)
- [Taming CATS: Controllable Automatic Text Simplification through Instruction Fine-Tuning with Control Tokens](https://mlsanity.com/paper/2604.01779/) (2026)
- [HULAT2 at MER-TRANS 2026: Governed Multi-Agent Simplification for Spanish Easy-to-Read Generation](https://mlsanity.com/paper/2607.02381/) (2026)
- [Reader Proficiency Shapes Layer-wise Surprisal Profiles](https://mlsanity.com/paper/2609.37688/) (2026)

## Abstract (arXiv)

Automatic text simplification (TS) aims to automate the process of rewriting text to make it easier for people to read. A pre-requisite for TS to be useful is that it should convey information that is consistent with the meaning of the original text. However, current TS evaluation protocols assess system outputs for simplicity and meaning preservation without regard for the document context in which output sentences occur and for how people understand them. In this work, we introduce a human evaluation framework to assess whether simplified texts preserve meaning using reading comprehension questions. With this framework, we conduct a thorough human evaluation of texts by humans and by nine automatic systems. Supervised systems that leverage pre-training knowledge achieve the highest scores on the reading comprehension (RC) tasks amongst the automatic controllable TS systems. However, even the best-performing supervised system struggles with at least 14% of the questions, marking them as "unanswerable'' based on simplified content. We further investigate how existing TS evaluation metrics and automatic question-answering systems approximate the human judgments we obtained.