# GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

arXiv: https://arxiv.org/abs/2311.08191 · HTML: https://mlsanity.com/paper/2311.08191/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Learning Latent Permutations with Gumbel-Sinkhorn Networks](https://mlsanity.com/paper/1802.08665/) (2018)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [Encode, Tag, Realize: High-Precision Text Editing](https://mlsanity.com/paper/1909.01187/) (2019)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Felix: Flexible Text Editing Through Tagging and Insertion](https://mlsanity.com/paper/2003.10687/) (2020)
- [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding](https://mlsanity.com/paper/2106.04970/) (2021)
- [EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start](https://mlsanity.com/paper/2205.12209/) (2022)
- [Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model](https://mlsanity.com/paper/2005.11849/) (2020)
- [Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation](https://mlsanity.com/paper/1808.09582/) (2018)
- [Grammatical Error Correction in Low-Resource Scenarios](https://mlsanity.com/paper/1910.00353/) (2019)
- [Data Weighted Training Strategies for Grammatical Error Correction](https://mlsanity.com/paper/2008.02976/) (2020)
- [Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction](https://mlsanity.com/paper/2010.03260/) (2020)
- [Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction](https://mlsanity.com/paper/2203.13064/) (2022)
- [The Unbearable Weight of Generating Artificial Errors for Grammatical Error Correction](https://mlsanity.com/paper/1907.08889/) (2019)

## Built on by (most cited first, 3 of 3)

- [Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule](https://mlsanity.com/paper/2311.11813/) (2023)
- [Detection-Correction Structure via General Language Model for Grammatical Error Correction](https://mlsanity.com/paper/2405.17804/) (2024)
- [Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correction](https://mlsanity.com/paper/2403.19283/) (2024)

## Abstract (arXiv)

Grammatical error correction (GEC) is an important NLP task that is currently usually solved with autoregressive sequence-to-sequence models. However, approaches of this class are inherently slow due to one-by-one token generation, so non-autoregressive alternatives are needed. In this work, we propose a novel non-autoregressive approach to GEC that decouples the architecture into a permutation network that outputs a self-attention weight matrix that can be used in beam search to find the best permutation of input tokens (with auxiliary {ins} tokens) and a decoder network based on a step-unrolled denoising autoencoder that fills in specific tokens. This allows us to find the token permutation after only one forward pass of the permutation network, avoiding autoregressive constructions. We show that the resulting network improves over previously known non-autoregressive methods for GEC and reaches the level of autoregressive methods that do not use language-specific synthetic data generation methods. Our results are supported by a comprehensive experimental validation on the ConLL-2014 and Write&Improve+LOCNESS datasets and an extensive ablation study that supports our architectural and algorithmic choices.