# LENS: A Learnable Evaluation Metric for Text Simplification

arXiv: https://arxiv.org/abs/2212.09739 · HTML: https://mlsanity.com/paper/2212.09739/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 106 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)

## Built on by (most cited first, 30 of 54)

- [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [Beyond the Chat: Executable and Verifiable Text-Editing with LLMs](https://mlsanity.com/paper/2309.15337/) (2023)
- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Texts](https://mlsanity.com/paper/2405.13179/) (2024)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment](https://mlsanity.com/paper/2305.14463/) (2023)
- [Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability](https://mlsanity.com/paper/2406.18365/) (2024)
- [Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles](https://mlsanity.com/paper/2408.08566/) (2024)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension](https://mlsanity.com/paper/2312.10126/) (2023)
- [APPLS: Evaluating Evaluation Metrics for Plain Language Summarization](https://mlsanity.com/paper/2305.14341/) (2023)
- [Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA](https://mlsanity.com/paper/2305.14458/) (2023)
- [Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification](https://mlsanity.com/paper/2310.08170/) (2023)
- [Improving Minimum Bayes Risk Decoding with Multi-Prompt](https://mlsanity.com/paper/2407.15343/) (2024)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)
- [Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation](https://mlsanity.com/paper/2308.06953/) (2023)
- [Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks](https://mlsanity.com/paper/2311.00508/) (2023)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Automatic and Human-AI Interactive Text Generation](https://mlsanity.com/paper/2310.03878/) (2023)
- [German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data](https://mlsanity.com/paper/2402.10675/) (2024)
- [An In-depth Evaluation of Large Language Models in Sentence Simplification with Error-based Human Assessment](https://mlsanity.com/paper/2403.04963/) (2024)
- [Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM](https://mlsanity.com/paper/2407.04067/) (2024)
- [OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMs](https://mlsanity.com/paper/2503.11858/) (2025)
- [Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification](https://mlsanity.com/paper/2504.09394/) (2025)
- [LLM-based Text Simplification and its Effect on User Comprehension and Cognitive Load](https://mlsanity.com/paper/2505.01980/) (2025)
- [Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual Disabilities](https://mlsanity.com/paper/2402.13094/) (2024)
- [REFeREE: A REference-FREE Model-Based Metric for Text Simplification](https://mlsanity.com/paper/2403.17640/) (2024)

## Abstract (arXiv)

Training learnable metrics using modern language models has recently emerged as a promising method for the automatic evaluation of machine translation. However, existing human evaluation datasets for text simplification have limited annotations that are based on unitary or outdated models, making them unsuitable for this approach. To address these issues, we introduce the SimpEval corpus that contains: SimpEval_past, comprising 12K human ratings on 2.4K simplifications of 24 past systems, and SimpEval_2022, a challenging simplification benchmark consisting of over 1K human ratings of 360 simplifications including GPT-3.5 generated text. Training on SimpEval, we present LENS, a Learnable Evaluation Metric for Text Simplification. Extensive empirical results show that LENS correlates much better with human judgment than existing metrics, paving the way for future progress in the evaluation of text simplification. We also introduce Rank and Rate, a human evaluation framework that rates simplifications from several models in a list-wise manner using an interactive interface, which ensures both consistency and accuracy in the evaluation process and is used to create the SimpEval datasets.