# Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation

arXiv: https://arxiv.org/abs/2404.03278 · HTML: https://mlsanity.com/paper/2404.03278/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [Controllable Sentence Simplification](https://mlsanity.com/paper/1910.02677/) (2019)
- [Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text](https://mlsanity.com/paper/2107.03444/) (2021)
- [MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics](https://mlsanity.com/paper/2010.03636/) (2020)
- [Elaborative Simplification: Content Addition and Explanation Generation in Text Simplification](https://mlsanity.com/paper/2010.10035/) (2020)
- [Document-Level Text Simplification: Dataset, Criteria and Baseline](https://mlsanity.com/paper/2110.05071/) (2021)
- [Dynamic Multi-Level Multi-Task Learning for Sentence Simplification](https://mlsanity.com/paper/1806.07304/) (2018)
- [FFCI: A Framework for Interpretable Automatic Evaluation of Summarization](https://mlsanity.com/paper/2011.13662/) (2020)
- [Simple and Effective Text Simplification Using Semantic and Neural Methods](https://mlsanity.com/paper/1810.05104/) (2018)
- [Discourse Level Factors for Sentence Deletion in Text Simplification](https://mlsanity.com/paper/1911.10384/) (2019)
- [Rethinking Automatic Evaluation in Sentence Simplification](https://mlsanity.com/paper/2104.07560/) (2021)
- [Transforming Complex Sentences into a Semantic Hierarchy](https://mlsanity.com/paper/1906.01038/) (2019)
- [Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification](https://mlsanity.com/paper/2310.08170/) (2023)
- [Sentence Simplification with Memory-Augmented Neural Networks](https://mlsanity.com/paper/1804.07445/) (2018)
- [A Sentence Simplification System for Improving Relation Extraction](https://mlsanity.com/paper/1703.09013/) (2017)
- [Deep Learning Approaches to Lexical Simplification: A Survey](https://mlsanity.com/paper/2305.12000/) (2023)

## Built on by (most cited first, 5 of 5)

- [Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM](https://mlsanity.com/paper/2407.04067/) (2024)
- [Text Adaptation to Plain Language and Easy Read via Automatic Post-Editing Cycles](https://mlsanity.com/paper/2509.11991/) (2025)
- [DETECT: Determining Ease and Textual Clarity of German Text Simplifications](https://mlsanity.com/paper/2510.22212/) (2025)
- [Readability Measures and Automatic Text Simplification: In the Search of a Construct](https://mlsanity.com/paper/2511.09536/) (2025)
- [HULAT2 at MER-TRANS 2026: Governed Multi-Agent Simplification for Spanish Easy-to-Read Generation](https://mlsanity.com/paper/2607.02381/) (2026)

## Abstract (arXiv)

Text simplification intends to make a text easier to read while preserving its core meaning. Intuitively and as shown in previous works, these two dimensions (simplification and meaning preservation) are often-times inversely correlated. An overly conservative text will fail to simplify sufficiently, whereas extreme simplification will degrade meaning preservation. Yet, popular evaluation metrics either aggregate meaning preservation and simplification into a single score (SARI, LENS), or target meaning preservation alone (BERTScore, QuestEval). Moreover, these metrics usually require a set of references and most previous work has only focused on sentence-level simplification. In this paper, we focus on the evaluation of document-level text simplification and compare existing models using distinct metrics for meaning preservation and simplification. We leverage existing metrics from similar tasks and introduce a reference-less metric variant for simplicity, showing that models are mostly biased towards either simplification or meaning preservation, seldom performing well on both dimensions. Making use of the fact that the metrics we use are all reference-less, we also investigate the performance of existing models when applied to unseen data (where reference simplifications are unavailable).