# An In-depth Evaluation of Large Language Models in Sentence Simplification with Error-based Human Assessment

arXiv: https://arxiv.org/abs/2403.04963 · HTML: https://mlsanity.com/paper/2403.04963/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://mlsanity.com/paper/2005.00481/) (2020)
- [Document-Level Machine Translation with Large Language Models](https://mlsanity.com/paper/2304.02210/) (2023)
- [BLEU is Not Suitable for the Evaluation of Text Simplification](https://mlsanity.com/paper/1810.05995/) (2018)
- [Controllable Sentence Simplification](https://mlsanity.com/paper/1910.02677/) (2019)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://mlsanity.com/paper/2212.09739/) (2022)
- [Controllable Text Simplification with Explicit Paraphrasing](https://mlsanity.com/paper/2010.11004/) (2020)
- [EASSE: Easier Automatic Sentence Simplification Evaluation](https://mlsanity.com/paper/1908.04567/) (2019)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://mlsanity.com/paper/1810.11193/) (2018)
- [Simple and Effective Text Simplification Using Semantic and Neural Methods](https://mlsanity.com/paper/1810.05104/) (2018)
- [Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification](https://mlsanity.com/paper/1904.02767/) (2019)
- [Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA](https://mlsanity.com/paper/2305.14458/) (2023)

## Built on by (most cited first, 4 of 4)

- [Policy-based Sentence Simplification: Replacing Parallel Corpora with LLM-as-a-Judge](https://mlsanity.com/paper/2512.06228/) (2025)
- [Simplify-This: A Comparative Analysis of Prompt-Based and Fine-Tuned LLMs](https://mlsanity.com/paper/2601.05794/) (2026)
- [Making Knowledge Accessible: Divergent Readability-Accuracy Strategies of Mistral and QWen in Biomedical Text Simplification](https://mlsanity.com/paper/2511.05080/) (2025)
- [Profiling German Text Simplification with Interpretable Model-Fingerprints](https://mlsanity.com/paper/2601.13050/) (2026)

## Abstract (arXiv)

Recent studies have used both automatic metrics and human evaluations to assess the simplification abilities of LLMs. However, the suitability of existing evaluation methodologies for LLMs remains in question. First, the suitability of current automatic metrics on LLMs' simplification evaluation is still uncertain. Second, current human evaluation approaches in sentence simplification often fall into two extremes: they are either too superficial, failing to offer a clear understanding of the models' performance, or overly detailed, making the annotation process complex and prone to inconsistency, which in turn affects the evaluation's reliability. To address these problems, this study provides in-depth insights into LLMs' performance while ensuring the reliability of the evaluation. We design an error-based human annotation framework to assess the LLMs' simplification capabilities. We select both closed-source and open-source LLMs, including GPT-4, Qwen2.5-72B, and Llama-3.2-3B. We believe that these models offer a representative selection across large, medium, and small sizes of LLMs. Results show that LLMs generally generate fewer erroneous simplification outputs compared to the previous state-of-the-art. However, LLMs have their limitations, as seen in GPT-4's and Qwen2.5-72B's struggle with lexical paraphrasing. Furthermore, we conduct meta-evaluations on widely used automatic metrics using our human annotations. We find that these metrics lack sufficient sensitivity to assess the overall high-quality simplifications, particularly those generated by high-performance LLMs.