# Investigating Text Simplification Evaluation

arXiv: https://arxiv.org/abs/2107.13662 · HTML: https://mlsanity.com/paper/2107.13662/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 28 Jul 2021.

## Builds on (most cited first, 13 of 13)

- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://mlsanity.com/paper/2005.00481/) (2020)
- [BLEU is Not Suitable for the Evaluation of Text Simplification](https://mlsanity.com/paper/1810.05995/) (2018)
- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://mlsanity.com/paper/1906.08104/) (2019)
- [Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction](https://mlsanity.com/paper/1906.09317/) (2019)
- [Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen](https://mlsanity.com/paper/2005.00701/) (2020)
- [Unsupervised Neural Text Simplification](https://mlsanity.com/paper/1810.07931/) (2018)
- [Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://mlsanity.com/paper/1810.11193/) (2018)
- [Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale](https://mlsanity.com/paper/2010.13588/) (2020)
- [Dynamic Multi-Level Multi-Task Learning for Sentence Simplification](https://mlsanity.com/paper/1806.07304/) (2018)
- [Sentence Simplification with Memory-Augmented Neural Networks](https://mlsanity.com/paper/1804.07445/) (2018)
- [The Daunting Task of Real-World Textual Style Transfer Auto-Evaluation](https://mlsanity.com/paper/1910.03747/) (2019)

## Built on by (most cited first, 5 of 5)

- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [Revisiting non-English Text Simplification: A Unified Multilingual Benchmark](https://mlsanity.com/paper/2305.15678/) (2023)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Taming CATS: Controllable Automatic Text Simplification through Instruction Fine-Tuning with Control Tokens](https://mlsanity.com/paper/2604.01779/) (2026)
- [Reinforcement Learning for improving Large Language Models' Catalan text simplification capabilities](https://mlsanity.com/paper/2609.04823/) (2026)

## Abstract (arXiv)

Modern text simplification (TS) heavily relies on the availability of gold standard data to build machine learning models. However, existing studies show that parallel TS corpora contain inaccurate simplifications and incorrect alignments. Additionally, evaluation is usually performed by using metrics such as BLEU or SARI to compare system output to the gold standard. A major limitation is that these metrics do not match human judgements and the performance on different datasets and linguistic phenomena vary greatly. Furthermore, our research shows that the test and training subsets of parallel datasets differ significantly. In this work, we investigate existing TS corpora, providing new insights that will motivate the improvement of existing state-of-the-art TS evaluation methods. Our contributions include the analysis of TS corpora based on existing modifications used for simplification and an empirical study on TS models performance by using better-distributed datasets. We demonstrate that by improving the distribution of TS datasets, we can build more robust TS models.