# Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction

arXiv: https://arxiv.org/abs/2106.01609 · HTML: https://mlsanity.com/paper/2106.01609/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data](https://mlsanity.com/paper/1903.00138/) (2019)
- [Non-Autoregressive Text Generation with Pre-trained Language Models](https://mlsanity.com/paper/2102.08220/) (2021)
- [Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study](https://mlsanity.com/paper/1807.01270/) (2018)
- [A Comprehensive Survey of Grammar Error Correction](https://mlsanity.com/paper/2005.06600/) (2020)
- [TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis](https://mlsanity.com/paper/2012.15639/) (2020)
- [An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation](https://mlsanity.com/paper/2003.04195/) (2020)
- [Chinese Grammatical Correction Using BERT-based Pre-trained Model](https://mlsanity.com/paper/2011.02093/) (2020)
- [Open Challenge for Correcting Errors of Speech Recognition Systems](https://mlsanity.com/paper/2001.03041/) (2020)

## Built on by (most cited first, 19 of 19)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2204.10994/) (2022)
- [General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining](https://mlsanity.com/paper/2203.10929/) (2022)
- [Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correction](https://mlsanity.com/paper/2210.10442/) (2022)
- [Rethinking Masked Language Modeling for Chinese Spelling Correction](https://mlsanity.com/paper/2305.17721/) (2023)
- [A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check](https://mlsanity.com/paper/2310.09119/) (2023)
- [Correct Like Humans: Progressive Learning Framework for Chinese Text Error Correction](https://mlsanity.com/paper/2306.17447/) (2023)
- [C-LLM: Learn to Check Chinese Spelling Errors Character by Character](https://mlsanity.com/paper/2406.16536/) (2024)
- [uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers](https://mlsanity.com/paper/2209.07068/) (2022)
- [Effidit: Your AI Writing Assistant](https://mlsanity.com/paper/2208.01815/) (2022)
- [From Spelling to Grammar: A New Framework for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2211.01625/) (2022)
- [An Error-Guided Correction Model for Chinese Spelling Error Correction](https://mlsanity.com/paper/2301.06323/) (2023)
- [FlaCGEC: A Chinese Grammatical Error Correction Dataset with Fine-grained Linguistic Annotation](https://mlsanity.com/paper/2311.04906/) (2023)
- [Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting](https://mlsanity.com/paper/2310.13321/) (2023)
- [ChatLang-8: An LLM-Based Synthetic Data Generation Framework for Grammatical Error Correction](https://mlsanity.com/paper/2406.03202/) (2024)
- [Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors](https://mlsanity.com/paper/2204.12052/) (2022)
- [An Adversarial Multi-Task Learning Method for Chinese Text Correction with Semantic Detection](https://mlsanity.com/paper/2306.16313/) (2023)
- [Exploring the Usage of Chinese Pinyin in Pretraining](https://mlsanity.com/paper/2310.04960/) (2023)
- [AxBERT: An Interpretable Chinese Spelling Correction Method Driven by Associative Knowledge Network](https://mlsanity.com/paper/2503.02255/) (2025)

## Abstract (arXiv)

We investigate the problem of Chinese Grammatical Error Correction (CGEC) and present a new framework named Tail-to-Tail (TtT) non-autoregressive sequence prediction to address the deep issues hidden in CGEC. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected based on the bidirectional context information, thus we employ a BERT-initialized Transformer Encoder as the backbone model to conduct information modeling and conveying. Considering that only relying on the same position substitution cannot handle the variable-length correction cases, various operations such substitution, deletion, insertion, and local paraphrasing are required jointly. Therefore, a Conditional Random Fields (CRF) layer is stacked on the up tail to conduct non-autoregressive sequence prediction by modeling the token dependencies. Since most tokens are correct and easily to be predicted/conveyed to the target, then the models may suffer from a severe class imbalance issue. To alleviate this problem, focal loss penalty strategies are integrated into the loss functions. Moreover, besides the typical fix-length error correction datasets, we also construct a variable-length corpus to conduct experiments. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of error Detection and Correction.