# Multi-teacher Distillation for Multilingual Spelling Correction

arXiv: https://arxiv.org/abs/2311.11518 · HTML: https://mlsanity.com/paper/2311.11518/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 22 of 22)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [Feature-map-level Online Adversarial Knowledge Distillation](https://mlsanity.com/paper/2002.01775/) (2020)
- [ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation](https://mlsanity.com/paper/2106.01597/) (2021)
- [GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors](https://mlsanity.com/paper/1911.12893/) (2019)
- [MTG: A Benchmark Suite for Multilingual Text Generation](https://mlsanity.com/paper/2108.07140/) (2021)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [Spelling Correction with Denoising Transformer](https://mlsanity.com/paper/2105.05977/) (2021)
- [Spelling Correction as a Foreign Language](https://mlsanity.com/paper/1705.07371/) (2017)
- [Contextual Multilingual Spellchecker for User Queries](https://mlsanity.com/paper/2305.01082/) (2023)

## Built on by (most cited first, 1 of 1)

- [Retrieval Augmented Spelling Correction for E-Commerce Applications](https://mlsanity.com/paper/2410.11655/) (2024)

## Abstract (arXiv)

Accurate spelling correction is a critical step in modern search interfaces, especially in an era of mobile devices and speech-to-text interfaces. For services that are deployed around the world, this poses a significant challenge for multilingual NLP: spelling errors need to be caught and corrected in all languages, and even in queries that use multiple languages. In this paper, we tackle this challenge using multi-teacher distillation. On our approach, a monolingual teacher model is trained for each language/locale, and these individual models are distilled into a single multilingual student model intended to serve all languages/locales. In experiments using open-source data as well as user data from a worldwide search service, we show that this leads to highly effective spelling correction models that can meet the tight latency requirements of deployed services.