mlsanityData to 7 Oct 2026

ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

arXiv 2303.13648 ↗First posted 15 Mar 2023cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted a lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim to evaluate ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g., GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT performs not as well as those baselines in terms of the automatic evaluation metrics (e.g., F0.5 score), particularly on long sentences. We inspect the outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically, it prefers to change the surface expression of certain phrases or sentence structure while maintaining grammatical correctness. Human evaluation quantitatively confirms this and suggests that ChatGPT produces less under-correction or mis-correction issues but more over-corrections. These results demonstrate that ChatGPT is severely under-estimated by the automatic evaluation metrics and could be a promising tool for GEC.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

15 of 22 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2023Mathematical Capabilities of ChatGPTcited by 263
  8. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
  9. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  10. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  11. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  12. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
3 more
  1. 2022Grammatical Error Correction: A Survey of the State of the Artcited by 43
  2. 2023Conversational Automated Program Repaircited by 41
  3. 2022Revisiting Grammatical Error Correction Evaluation and Beyondcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 39; 2024: 25; 2025: 12; 2026: 1.

77 papers.

Sort:

By area:

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  3. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  4. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  5. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
  6. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  7. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  8. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 67
  9. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  10. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  11. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  12. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 56
38 more
  1. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  2. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 53
  3. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  4. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  5. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  6. 2023Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Modelscited by 42
  7. 2024Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversationscited by 36
  8. 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34
  9. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  10. 2023On the application of Large Language Models for language teaching and assessment technologycited by 26
  11. 2024Prompting open-source and commercial language models for grammatical error correction of English learner textcited by 24
  12. 2024ASR Error Correction using Large Language Modelscited by 23
  13. 2024How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHOcited by 19
  14. 2024GPT-3.5 for Grammatical Error Correctioncited by 18
  15. 2023Revisiting the Reliability of Psychological Scales on Large Language Modelscited by 16
  16. 2025Whitened CLIP as a Likelihood Surrogate of Images and Captionscited by 16
  17. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  18. 2024mEdIT: Multilingual Text Editing via Instruction Tuningcited by 13
  19. 2023Generating Valid and Natural Adversarial Examples with Large Language Modelscited by 12
  20. 2023Beyond English: Evaluating LLMs for Arabic Grammatical Error Correctioncited by 12
  21. 2024C-LLM: Learn to Check Chinese Spelling Errors Character by Charactercited by 12
  22. 2023NaSGEC: a Multi-Domain Chinese Grammatical Error Correction Dataset from Native Speaker Textscited by 9
  23. 2025Enhancing Text Editing for Grammatical Error Correction: Arabic as a Case Studycited by 9
  24. 2023Multimodal Large Language Model for Visual Navigationcited by 8
  25. 2023Assessing Translation capabilities of Large Language Models involving English and Indian Languagescited by 8
  26. 2024Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Modelscited by 8
  27. 2023Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correctioncited by 7
  28. 2023RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbationcited by 6
  29. 2025Opportunities and Challenges of LLMs in Education: An NLP Perspectivecited by 6
  30. 2023Reducing Sequence Length by Predicting Edit Operations with Large Language Modelscited by 5
  31. 2023Validating Multimedia Content Moderation Software via Semantic Fusioncited by 5
  32. 2023Evaluating GPT-3.5 and GPT-4 on Grammatical Error Correction for Brazilian Portuguesecited by 5
  33. 2023Evaluating the Capability of Large-scale Language Models on Chinese Grammatical Error Correction Taskcited by 5
  34. 2023ChatGPT for Arabic Grammatical Error Correctioncited by 5
  35. 2024Grammaticality Representation in ChatGPT as Compared to Linguists and Laypeoplecited by 5
  36. 2023Minimum Bayes' Risk Decoding for System Combination of Grammatical Error Correction Systemscited by 4
  37. 2024Spivavtor: An Instruction Tuned Ukrainian Text Editing Modelcited by 4
  38. 2024Proofread: Fixes All Errors with One Tapcited by 4

50 most cited of 77 All 77 (JSON)