# A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages

arXiv: https://arxiv.org/abs/2308.09435 · HTML: https://mlsanity.com/paper/2308.09435/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Comparison of Diverse Decoding Methods from Conditional Language Models](https://mlsanity.com/paper/1906.06362/) (2019)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2010.15925/) (2020)
- [NeuSpell: A Neural Spelling Correction Toolkit](https://mlsanity.com/paper/2010.11085/) (2020)
- [GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors](https://mlsanity.com/paper/1911.12893/) (2019)
- [TAPE: Assessing Few-shot Russian Language Understanding](https://mlsanity.com/paper/2210.12813/) (2022)
- [Spelling Correction with Denoising Transformer](https://mlsanity.com/paper/2105.05977/) (2021)
- [CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription](https://mlsanity.com/paper/2107.01091/) (2021)
- [Contextual Text Denoising with Masked Language Models](https://mlsanity.com/paper/1910.14080/) (2019)
- [Spelling Correction as a Foreign Language](https://mlsanity.com/paper/1705.07371/) (2017)
- [Zero-Shot Paraphrase Generation with Multilingual Language Models](https://mlsanity.com/paper/1911.03597/) (2019)
- [BSpell: A CNN-Blended BERT Based Bangla Spell Checker](https://mlsanity.com/paper/2208.09709/) (2022)

## Built on by (most cited first, 5 of 5)

- [The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design](https://mlsanity.com/paper/2408.12503/) (2024)
- [Detecting Spelling and Grammatical Anomalies in Russian Poetry Texts](https://mlsanity.com/paper/2505.04507/) (2025)
- [Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems](https://mlsanity.com/paper/2505.22771/) (2025)
- [LMSpell: Spell Correction with Pre-Trained Language Models](https://mlsanity.com/paper/2512.05414/) (2025)
- [OpenAutoNLU: Open Source AutoML Library for NLU](https://mlsanity.com/paper/2603.01824/) (2026)

## Abstract (arXiv)

Modern large language models demonstrate impressive capabilities in text generation and generalization. However, they often struggle with solving text editing tasks, particularly when it comes to correcting spelling errors and mistypings. In this paper, we present a methodology for generative spelling correction (SC), which was tested on English and Russian languages and potentially can be extended to any language with minor changes. Our research mainly focuses on exploring natural spelling errors and mistypings in texts and studying the ways those errors can be emulated in correct sentences to effectively enrich generative models' pre-train procedure. We investigate the impact of such emulations and the models' abilities across different text domains. In this work, we investigate two spelling corruption techniques: 1) first one mimics human behavior when making a mistake through leveraging statistics of errors from particular dataset and 2) second adds the most common spelling errors, keyboard miss clicks, and some heuristics within the texts. We conducted experiments employing various corruption strategies, models' architectures and sizes on the pre-training and fine-tuning stages and evaluated the models using single-domain and multi-domain test sets. As a practical outcome of our work, we introduce SAGE(Spell checking via Augmentation and Generative distribution Emulation). It is a library for automatic generative SC that includes a family of pre-trained generative models and built-in augmentation algorithms.