# Misspelling Correction with Pre-trained Contextual Language Model

arXiv: https://arxiv.org/abs/2101.03204 · HTML: https://mlsanity.com/paper/2101.03204/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 8 Jan 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study](https://mlsanity.com/paper/1807.01270/) (2018)
- [Robsut Wrod Reocginiton via semi-Character Recurrent Neural Network](https://mlsanity.com/paper/1608.02214/) (2016)
- [Spelling Error Correction Using a Nested RNN Model and Pseudo Training Data](https://mlsanity.com/paper/1811.00238/) (2018)
- [A context sensitive real-time Spell Checker with language adaptability](https://mlsanity.com/paper/1910.11242/) (2019)

## Built on by (most cited first, 9 of 9)

- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT](https://mlsanity.com/paper/2308.06294/) (2023)
- [BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks](https://mlsanity.com/paper/2106.01452/) (2021)
- [Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling](https://mlsanity.com/paper/2406.15527/) (2024)
- [ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study](https://mlsanity.com/paper/2401.06513/) (2024)
- [Persian Typographical Error Type Detection Using Deep Neural Networks on Algorithmically-Generated Misspellings](https://mlsanity.com/paper/2305.11731/) (2023)
- [Automatic Real-word Error Correction in Persian Text](https://mlsanity.com/paper/2407.14795/) (2024)
- [A Comprehensive Approach to Misspelling Correction with BERT and Levenshtein Distance](https://mlsanity.com/paper/2407.17383/) (2024)
- [Improving the quality of Persian clinical text with a novel spelling correction system](https://mlsanity.com/paper/2408.03622/) (2024)

## Abstract (arXiv)

Spelling irregularities, known now as spelling mistakes, have been found for several centuries. As humans, we are able to understand most of the misspelled words based on their location in the sentence, perceived pronunciation, and context. Unlike humans, computer systems do not possess the convenient auto complete functionality of which human brains are capable. While many programs provide spelling correction functionality, many systems do not take context into account. Moreover, Artificial Intelligence systems function in the way they are trained on. With many current Natural Language Processing (NLP) systems trained on grammatically correct text data, many are vulnerable against adversarial examples, yet correctly spelled text processing is crucial for learning. In this paper, we investigate how spelling errors can be corrected in context, with a pre-trained language model BERT. We present two experiments, based on BERT and the edit distance algorithm, for ranking and selecting candidate corrections. The results of our experiments demonstrated that when combined properly, contextual word embeddings of BERT and edit distance are capable of effectively correcting spelling errors.