# Spelling Correction with Denoising Transformer

arXiv: https://arxiv.org/abs/2105.05977 · HTML: https://mlsanity.com/paper/2105.05977/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Gmail Smart Compose: Real-Time Assisted Writing](https://mlsanity.com/paper/1906.00080/) (2019)
- [MaskGAN: Better Text Generation via Filling in the______](https://mlsanity.com/paper/1801.07736/) (2018)
- [Tensor2Tensor for Neural Machine Translation](https://mlsanity.com/paper/1803.07416/) (2018)
- [A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning](https://mlsanity.com/paper/1907.01256/) (2019)
- [LABR: A Large Scale Arabic Sentiment Analysis Benchmark](https://mlsanity.com/paper/1411.6718/) (2014)
- [Complexity of Word Collocation Networks: A Preliminary Structural Analysis](https://mlsanity.com/paper/1310.5111/) (2013)
- [A Focus on Neural Machine Translation for African Languages](https://mlsanity.com/paper/1906.05685/) (2019)
- [Denoising based Sequence-to-Sequence Pre-training for Text Generation](https://mlsanity.com/paper/1908.08206/) (2019)
- [Neural Networks for Text Correction and Completion in Keyboard Decoding](https://mlsanity.com/paper/1709.06429/) (2017)
- [Artificial Error Generation with Machine Translation and Syntactic Patterns](https://mlsanity.com/paper/1707.05236/) (2017)
- [Spelling Correction as a Foreign Language](https://mlsanity.com/paper/1705.07371/) (2017)

## Built on by (most cited first, 9 of 9)

- [Correcting diacritics and typos with a ByT5 transformer model](https://mlsanity.com/paper/2201.13242/) (2022)
- [A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages](https://mlsanity.com/paper/2308.09435/) (2023)
- [Correcting Arabic Soft Spelling Mistakes using BiLSTM-based Machine Learning](https://mlsanity.com/paper/2108.01141/) (2021)
- [DPCSpell: A Transformer-based Detector-Purificator-Corrector Framework for Spelling Error Correction of Bangla and Resource Scarce Indic Languages](https://mlsanity.com/paper/2211.03730/) (2022)
- [Contextual Multilingual Spellchecker for User Queries](https://mlsanity.com/paper/2305.01082/) (2023)
- [Multi-teacher Distillation for Multilingual Spelling Correction](https://mlsanity.com/paper/2311.11518/) (2023)
- [Contextual Spelling Correction with Language Model for Low-resource Setting](https://mlsanity.com/paper/2404.18072/) (2024)
- [A Comprehensive Approach to Misspelling Correction with BERT and Levenshtein Distance](https://mlsanity.com/paper/2407.17383/) (2024)
- [Spelling Correction in Healthcare Query-Answer Systems: Methods, Retrieval Impact, and Empirical Evaluation](https://mlsanity.com/paper/2603.19249/) (2026)

## Abstract (arXiv)

We present a novel method of performing spelling correction on short input strings, such as search queries or individual words. At its core lies a procedure for generating artificial typos which closely follow the error patterns manifested by humans. This procedure is used to train the production spelling correction model based on a transformer architecture. This model is currently served in the HubSpot product search. We show that our approach to typo generation is superior to the widespread practice of adding noise, which ignores human patterns. We also demonstrate how our approach may be extended to resource-scarce settings and train spelling correction models for Arabic, Greek, Russian, and Setswana languages, without using any labeled data.