mlsanityData to 7 Oct 2026

LM-Critic: Language Models for Unsupervised Grammatical Error Correction

arXiv 2109.06822 ↗First posted 14 Sep 2021cs.CL cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Training a model for grammatical error correction (GEC) requires a set of labeled ungrammatical / grammatical sentence pairs, but manually annotating such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has demonstrated strong results on learning to repair a broken program without any labeled examples, but this relies on a perfect critic (e.g., a compiler) that returns whether an example is valid or not, which does not exist for the GEC task. In this work, we show how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations. We apply this LM-Critic and BIFI along with a large set of unlabeled sentences to bootstrap realistic ungrammatical / grammatical pairs for training a corrector. We evaluate our approach on GEC datasets across multiple domains (CoNLL-2014, BEA-2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existing methods in both the unsupervised setting (+7.7 F0.5) and the supervised setting (+0.5 F0.5).

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

35 of 62 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
23 more
  1. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  2. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  3. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
  4. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  5. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  6. 2019Parallel Iterative Edit Models for Local Sequence Transductioncited by 81
  7. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  8. 2020Robust Encodings: A Framework for Combating Adversarial Typoscited by 58
  9. 2018Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Taskcited by 56
  10. 2019An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correctioncited by 55
  11. 2019Corpora Generation for Grammatical Error Correctioncited by 53
  12. 2019Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Datacited by 50
  13. 2018Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!cited by 43
  14. 2018A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correctioncited by 36
  15. 2019Grammatical Error Correction in Low-Resource Scenarioscited by 28
  16. 2020Unsupervised Parsing via Constituency Testscited by 23
  17. 2016Neural Language Correction with Character-Based Attentioncited by 18
  18. 2016There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correctioncited by 18
  19. 2017A Nested Attention Neural Hybrid Model for Grammatical Error Correctioncited by 12
  20. 2019TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Textscited by 10
  21. 2019Learning to combine Grammatical Error Correctionscited by 9
  22. 2019Neural Grammatical Error Correction with Finite State Transducerscited by 6
  23. 2019Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models --- Is Single-Corpus Evaluation Enough?cited by 6

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 10; 2024: 13; 2025: 4.

33 papers.

Sort:

By area:

  1. 2022Generating Sequences by Learning to Self-Correctcited by 271
  2. 2022Grammatical Error Correction: A Survey of the State of the Artcited by 43
  3. 2023Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methodscited by 37
  4. 2022SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parsercited by 32
  5. 2024Backtracking Improves Generation Safetycited by 31
  6. 2023On the application of Large Language Models for language teaching and assessment technologycited by 26
  7. 2024Prompting open-source and commercial language models for grammatical error correction of English learner textcited by 24
  8. 2023Small Language Models Improve Giants by Rewriting Their Outputscited by 18
  9. 2024GPT-3.5 for Grammatical Error Correctioncited by 18
  10. 2024Taxonomy-Guided Zero-Shot Recommendations with LLMscited by 15
  11. 2024ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pagescited by 10
  12. 2022uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkerscited by 9
21 more
  1. 2022Converge to the Truth: Factual Error Correction via Iterative Constrained Editingcited by 9
  2. 2022A Simple, Yet Effective Approach to Finding Biases in Code Generationcited by 8
  3. 2023Improving Seq2Seq Grammatical Error Correction via Decoding Interventionscited by 8
  4. 2023Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correctioncited by 7
  5. 2024NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Modelcited by 7
  6. 2024Improving Grammatical Error Correction via Contextual Data Augmentationcited by 6
  7. 2024EDEN: Empathetic Dialogues for English learningcited by 5
  8. 2023Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniquescited by 3
  9. 2024ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitutioncited by 3
  10. 2024Revisiting Meta-evaluation for Grammatical Error Correctioncited by 3
  11. 2023MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correctioncited by 2
  12. 2023Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmentingcited by 2
  13. 2024DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewritingcited by 2
  14. 2025Efficient Real-time Refinement of Language Model Text Generationcited by 2
  15. 2025Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Modelscited by 2
  16. 2025Leveraging What's Overfixed: Post-Correction via LLM Grammatical Error Overcorrectioncited by 2
  17. 2023A BERT-based Unsupervised Grammatical Error Correction Frameworkcited by 1
  18. 2023HTEC: Human Transcription Error Correctioncited by 1
  19. 2024LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correctioncited by 1
  20. 2024Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detectioncited by 0
  21. 2025Time Is Effort: Estimating Human Post-Editing Time for Grammar Error Correction Tool Evaluationcited by 0