# Text Generation with Text-Editing Models

arXiv: https://arxiv.org/abs/2206.07043 · HTML: https://mlsanity.com/paper/2206.07043/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Encode, Tag, Realize: High-Precision Text Editing](https://mlsanity.com/paper/1909.01187/) (2019)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://mlsanity.com/paper/1906.08104/) (2019)
- [Felix: Flexible Text Editing Through Tagging and Insertion](https://mlsanity.com/paper/2003.10687/) (2020)
- [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [Improving Multilingual Models with Language-Clustered Vocabularies](https://mlsanity.com/paper/2010.12777/) (2020)
- [Query Resolution for Conversational Search with Limited Supervision](https://mlsanity.com/paper/2005.11723/) (2020)
- [LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2105.08206/) (2021)
- [EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start](https://mlsanity.com/paper/2205.12209/) (2022)
- [Incomplete Utterance Rewriting as Semantic Segmentation](https://mlsanity.com/paper/2009.13166/) (2020)
- [Grammatical Error Correction in Low-Resource Scenarios](https://mlsanity.com/paper/1910.00353/) (2019)
- [Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction](https://mlsanity.com/paper/2010.03260/) (2020)
- [UA-GEC: Grammatical Error Correction and Fluency Corpus for the Ukrainian Language](https://mlsanity.com/paper/2103.16997/) (2021)
- [Data-to-Text Generation with Iterative Text Editing](https://mlsanity.com/paper/2011.01694/) (2020)
- [Character Transformations for Non-Autoregressive GEC Tagging](https://mlsanity.com/paper/2111.09280/) (2021)

## Built on by (most cited first, 21 of 21)

- [RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://mlsanity.com/paper/2305.08844/) (2023)
- [Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions](https://mlsanity.com/paper/2209.15565/) (2022)
- [On Improving Summarization Factual Consistency from Natural Language Feedback](https://mlsanity.com/paper/2212.09968/) (2022)
- [On Evaluating and Mitigating Gender Biases in Multilingual Settings](https://mlsanity.com/paper/2307.01503/) (2023)
- [Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language](https://mlsanity.com/paper/2311.14543/) (2023)
- [Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation](https://mlsanity.com/paper/2305.18893/) (2023)
- [SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages](https://mlsanity.com/paper/2305.19204/) (2023)
- [Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Text](https://mlsanity.com/paper/2305.16444/) (2023)
- [Non-autoregressive Text Editing with Copy-aware Latent Alignments](https://mlsanity.com/paper/2310.07821/) (2023)
- [Towards an On-device Agent for Text Rewriting](https://mlsanity.com/paper/2308.11807/) (2023)
- [Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions](https://mlsanity.com/paper/2402.13647/) (2024)
- [Proofread: Fixes All Errors with One Tap](https://mlsanity.com/paper/2406.04523/) (2024)
- [COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models](https://mlsanity.com/paper/2402.14889/) (2024)
- [Online Training of Large Language Models: Learn while chatting](https://mlsanity.com/paper/2403.04790/) (2024)
- [Query Rewriting for Effective Misinformation Discovery](https://mlsanity.com/paper/2210.07467/) (2022)
- [CLSE: Corpus of Linguistically Significant Entities](https://mlsanity.com/paper/2211.02423/) (2022)
- [Text Editing as Imitation Game](https://mlsanity.com/paper/2210.12276/) (2022)
- [Bidirectional Transformer Reranker for Grammatical Error Correction](https://mlsanity.com/paper/2305.13000/) (2023)
- [Toward Interactive Dictation](https://mlsanity.com/paper/2307.04008/) (2023)
- [Layout-Aware Text Editing for Efficient Transformation of Academic PDFs to Markdown](https://mlsanity.com/paper/2512.18115/) (2025)
- [Spelling Correction in Healthcare Query-Answer Systems: Methods, Retrieval Impact, and Empirical Evaluation](https://mlsanity.com/paper/2603.19249/) (2026)

## Abstract (arXiv)

Text-editing models have recently become a prominent alternative to seq2seq models for monolingual text-generation tasks such as grammatical error correction, simplification, and style transfer. These tasks share a common trait - they exhibit a large amount of textual overlap between the source and target texts. Text-editing models take advantage of this observation and learn to generate the output by predicting edit operations applied to the source sequence. In contrast, seq2seq models generate outputs word-by-word from scratch thus making them slow at inference time. Text-editing models provide several benefits over seq2seq models including faster inference speed, higher sample efficiency, and better control and interpretability of the outputs. This tutorial provides a comprehensive overview of text-editing models and current state-of-the-art approaches, and analyzes their pros and cons. We discuss challenges related to productionization and how these models can be used to mitigate hallucination and bias, both pressing challenges in the field of text generation.