# LM-Critic: Language Models for Unsupervised Grammatical Error Correction

arXiv: https://arxiv.org/abs/2109.06822 · HTML: https://mlsanity.com/paper/2109.06822/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Combating Adversarial Misspellings with Robust Word Recognition](https://mlsanity.com/paper/1905.11268/) (2019)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [Robust Encodings: A Framework for Combating Adversarial Typos](https://mlsanity.com/paper/2005.01229/) (2020)
- [Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task](https://mlsanity.com/paper/1804.05940/) (2018)
- [An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction](https://mlsanity.com/paper/1909.00502/) (2019)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)
- [Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data](https://mlsanity.com/paper/1903.00138/) (2019)
- [Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!](https://mlsanity.com/paper/1809.08731/) (2018)
- [A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction](https://mlsanity.com/paper/1801.08831/) (2018)
- [Grammatical Error Correction in Low-Resource Scenarios](https://mlsanity.com/paper/1910.00353/) (2019)
- [Unsupervised Parsing via Constituency Tests](https://mlsanity.com/paper/2010.03146/) (2020)
- [Neural Language Correction with Character-Based Attention](https://mlsanity.com/paper/1603.09727/) (2016)
- [There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction](https://mlsanity.com/paper/1610.02124/) (2016)

## Built on by (most cited first, 30 of 33)

- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Backtracking Improves Generation Safety](https://mlsanity.com/paper/2409.14586/) (2024)
- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [Prompting open-source and commercial language models for grammatical error correction of English learner text](https://mlsanity.com/paper/2401.07702/) (2024)
- [Small Language Models Improve Giants by Rewriting Their Outputs](https://mlsanity.com/paper/2305.13514/) (2023)
- [GPT-3.5 for Grammatical Error Correction](https://mlsanity.com/paper/2405.08469/) (2024)
- [Taxonomy-Guided Zero-Shot Recommendations with LLMs](https://mlsanity.com/paper/2406.14043/) (2024)
- [ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages](https://mlsanity.com/paper/2403.17859/) (2024)
- [uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers](https://mlsanity.com/paper/2209.07068/) (2022)
- [Converge to the Truth: Factual Error Correction via Iterative Constrained Editing](https://mlsanity.com/paper/2211.12130/) (2022)
- [A Simple, Yet Effective Approach to Finding Biases in Code Generation](https://mlsanity.com/paper/2211.00609/) (2022)
- [Improving Seq2Seq Grammatical Error Correction via Decoding Interventions](https://mlsanity.com/paper/2310.14534/) (2023)
- [Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction](https://mlsanity.com/paper/2303.14342/) (2023)
- [NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model](https://mlsanity.com/paper/2411.05945/) (2024)
- [Improving Grammatical Error Correction via Contextual Data Augmentation](https://mlsanity.com/paper/2406.17456/) (2024)
- [EDEN: Empathetic Dialogues for English learning](https://mlsanity.com/paper/2406.17982/) (2024)
- [Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniques](https://mlsanity.com/paper/2310.08101/) (2023)
- [ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution](https://mlsanity.com/paper/2401.11356/) (2024)
- [Revisiting Meta-evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2403.02674/) (2024)
- [MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction](https://mlsanity.com/paper/2310.11671/) (2023)
- [Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting](https://mlsanity.com/paper/2310.13321/) (2023)
- [DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting](https://mlsanity.com/paper/2406.19650/) (2024)
- [Efficient Real-time Refinement of Language Model Text Generation](https://mlsanity.com/paper/2501.07824/) (2025)
- [Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Models](https://mlsanity.com/paper/2505.06004/) (2025)
- [Leveraging What's Overfixed: Post-Correction via LLM Grammatical Error Overcorrection](https://mlsanity.com/paper/2509.20811/) (2025)
- [A BERT-based Unsupervised Grammatical Error Correction Framework](https://mlsanity.com/paper/2303.17367/) (2023)
- [HTEC: Human Transcription Error Correction](https://mlsanity.com/paper/2309.10089/) (2023)

## Abstract (arXiv)

Training a model for grammatical error correction (GEC) requires a set of labeled ungrammatical / grammatical sentence pairs, but manually annotating such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has demonstrated strong results on learning to repair a broken program without any labeled examples, but this relies on a perfect critic (e.g., a compiler) that returns whether an example is valid or not, which does not exist for the GEC task. In this work, we show how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations. We apply this LM-Critic and BIFI along with a large set of unlabeled sentences to bootstrap realistic ungrammatical / grammatical pairs for training a corrector. We evaluate our approach on GEC datasets across multiple domains (CoNLL-2014, BEA-2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existing methods in both the unsupervised setting (+7.7 F0.5) and the supervised setting (+0.5 F0.5).