# EditEval: An Instruction-Based Benchmark for Text Improvements

arXiv: https://arxiv.org/abs/2209.13331 · HTML: https://mlsanity.com/paper/2209.13331/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 68 references that are in the map.
- First posted on arXiv 27 Sep 2022.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://mlsanity.com/paper/2005.00481/) (2020)
- [Encode, Tag, Realize: High-Precision Text Editing](https://mlsanity.com/paper/1909.01187/) (2019)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [PEER: A Collaborative Language Model](https://mlsanity.com/paper/2208.11663/) (2022)
- [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018)
- [Understanding Iterative Revision from Human-Written Text](https://mlsanity.com/paper/2203.03802/) (2022)
- [EASSE: Easier Automatic Sentence Simplification Evaluation](https://mlsanity.com/paper/1908.04567/) (2019)
- [Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!](https://mlsanity.com/paper/1809.08731/) (2018)
- [FRUIT: Faithfully Reflecting Updated Information in Text](https://mlsanity.com/paper/2112.08634/) (2021)
- [A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction](https://mlsanity.com/paper/1801.08831/) (2018)
- [Semantic Structural Evaluation for Text Simplification](https://mlsanity.com/paper/1810.05022/) (2018)
- [Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://mlsanity.com/paper/1810.11193/) (2018)
- [ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation](https://mlsanity.com/paper/2101.08382/) (2021)
- [Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study](https://mlsanity.com/paper/1807.01270/) (2018)
- [NewsEdits: A News Article Revision Dataset and a Document-Level Reasoning Challenge](https://mlsanity.com/paper/2206.07106/) (2022)
- [BERTTune: Fine-Tuning Neural Machine Translation with BERTScore](https://mlsanity.com/paper/2106.02208/) (2021)

## Built on by (most cited first, 30 of 30)

- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [CoEdIT: Text Editing by Task-Specific Instruction Tuning](https://mlsanity.com/paper/2305.09857/) (2023)
- [RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting](https://mlsanity.com/paper/2305.15685/) (2023)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance](https://mlsanity.com/paper/2305.13225/) (2023)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language](https://mlsanity.com/paper/2311.14543/) (2023)
- [Tur[k]ingBench: A Challenge Benchmark for Web Agents](https://mlsanity.com/paper/2403.11905/) (2024)
- [CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions](https://mlsanity.com/paper/2403.00241/) (2024)
- [SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages](https://mlsanity.com/paper/2305.19204/) (2023)
- [KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions](https://mlsanity.com/paper/2403.03866/) (2024)
- [Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning](https://mlsanity.com/paper/2410.08081/) (2024)
- [EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices](https://mlsanity.com/paper/2507.01438/) (2025)
- [LLMs Corrupt Your Documents When You Delegate](https://mlsanity.com/paper/2604.15597/) (2026)
- [Towards an On-device Agent for Text Rewriting](https://mlsanity.com/paper/2308.11807/) (2023)
- [XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updates](https://mlsanity.com/paper/2309.11063/) (2023)
- [DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection](https://mlsanity.com/paper/2310.16776/) (2023)
- [Identifying Reliable Evaluation Metrics for Scientific Text Revision](https://mlsanity.com/paper/2506.04772/) (2025)
- [WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia](https://mlsanity.com/paper/2507.03373/) (2025)
- [Automatic and Human-AI Interactive Text Generation](https://mlsanity.com/paper/2310.03878/) (2023)
- [InfoSync: Information Synchronization across Multilingual Semi-structured Tables](https://mlsanity.com/paper/2307.03313/) (2023)
- [Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction](https://mlsanity.com/paper/2403.17540/) (2024)
- [GradEscape: A Gradient-Based Evader Against AI-Generated Text Detectors](https://mlsanity.com/paper/2506.08188/) (2025)
- [CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes](https://mlsanity.com/paper/2606.31435/) (2026)
- [G-SPEED: General SParse Efficient Editing MoDel](https://mlsanity.com/paper/2310.10480/) (2023)
- [WRAVAL -- WRiting Assist eVALuation](https://mlsanity.com/paper/2601.03268/) (2025)
- [Intention-Adaptive LLM Fine-Tuning for Text Revision Generation](https://mlsanity.com/paper/2602.00477/) (2026)
- [Controllable Narrative Rendering for Enhanced Assisted Writing](https://mlsanity.com/paper/2607.00009/) (2026)

## Abstract (arXiv)

Evaluation of text generation to date has primarily focused on content created sequentially, rather than improvements on a piece of text. Writing, however, is naturally an iterative and incremental process that requires expertise in different modular skills such as fixing outdated information or making the style more consistent. Even so, comprehensive evaluation of a model's capacity to perform these skills and the ability to edit remains sparse. This work presents EditEval: An instruction-based, benchmark and evaluation suite that leverages high-quality existing and new datasets for automatic evaluation of editing capabilities such as making text more cohesive and paraphrasing. We evaluate several pre-trained models, which shows that InstructGPT and PEER perform the best, but that most baselines fall below the supervised SOTA, particularly when neutralizing and updating information. Our analysis also shows that commonly used metrics for editing tasks do not always correlate well, and that optimization for prompts with the highest performance does not necessarily entail the strongest robustness to different models. Through the release of this benchmark and a publicly available leaderboard challenge, we hope to unlock future research in developing models capable of iterative and more controllable editing.