# Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction

arXiv: https://arxiv.org/abs/2211.13252 · HTML: https://mlsanity.com/paper/2211.13252/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [Fine-Grained Semantically Aligned Vision-Language Pre-Training](https://mlsanity.com/paper/2208.02515/) (2022)
- [Felix: Flexible Text Editing Through Tagging and Insertion](https://mlsanity.com/paper/2003.10687/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [ASR Error Correction and Domain Adaptation Using Machine Translation](https://mlsanity.com/paper/2003.07692/) (2020)
- [FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition](https://mlsanity.com/paper/2105.03842/) (2021)
- [FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition](https://mlsanity.com/paper/2109.14420/) (2021)
- [Improving Readability for Automatic Speech Recognition Transcription](https://mlsanity.com/paper/2004.04438/) (2020)
- [Highway Long Short-Term Memory RNNs for Distant Speech Recognition](https://mlsanity.com/paper/1510.08983/) (2015)
- [Sequence-to-Action: Grammatical Error Correction with Action Guided Sequence Generation](https://mlsanity.com/paper/2205.10884/) (2022)
- [Concept Pointer Network for Abstractive Summarization](https://mlsanity.com/paper/1910.08486/) (2019)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [Cross Copy Network for Dialogue Generation](https://mlsanity.com/paper/2010.11539/) (2020)
- [Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency](https://mlsanity.com/paper/1805.04033/) (2018)

## Built on by (most cited first, 10 of 10)

- [Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration](https://mlsanity.com/paper/2310.09241/) (2023)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognition](https://mlsanity.com/paper/2212.01039/) (2022)
- [Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction](https://mlsanity.com/paper/2505.24347/) (2025)
- [Adapting an Unadaptable ASR System](https://mlsanity.com/paper/2306.01208/) (2023)
- [Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition](https://mlsanity.com/paper/2407.12817/) (2024)
- [Xinyu: An Efficient LLM-based System for Commentary Generation](https://mlsanity.com/paper/2408.11609/) (2024)
- [Optimized Tokenization for Transcribed Error Correction](https://mlsanity.com/paper/2310.10704/) (2023)
- [Auto Review: Second Stage Error Detection for Highly Accurate Information Extraction from Phone Conversations](https://mlsanity.com/paper/2506.05400/) (2025)

## Abstract (arXiv)

Text error correction aims to correct the errors in text sequences such as those typed by humans or generated by speech recognition models. Previous error correction methods usually take the source (incorrect) sentence as encoder input and generate the target (correct) sentence through the decoder. Since the error rate of the incorrect sentence is usually low (e.g., 10%), the correction model can only learn to correct on limited error tokens but trivially copy on most tokens (correct tokens), which harms the effective training of error correction. In this paper, we argue that the correct tokens should be better utilized to facilitate effective training and then propose a simple yet effective masking strategy to achieve this goal. Specifically, we randomly mask out a part of the correct tokens in the source sentence and let the model learn to not only correct the original error tokens but also predict the masked tokens based on their context information. Our method enjoys several advantages: 1) it alleviates trivial copy; 2) it leverages effective training signals from correct tokens; 3) it is a plug-and-play module and can be applied to different models and tasks. Experiments on spelling error correction and speech recognition error correction on Mandarin datasets and grammar error correction on English datasets with both autoregressive and non-autoregressive generation models show that our method improves the correction accuracy consistently.