mlsanityData to 7 Oct 2026

Efficient and Interpretable Grammatical Error Correction with Mixture of Experts

arXiv 2410.23507 ↗First posted 30 Oct 2024cs.CLcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Error type information has been widely used to improve the performance of grammatical error correction (GEC) models, whether for generating corrections, re-ranking them, or combining GEC models. Combining GEC models that have complementary strengths in correcting different error types is very effective in producing better corrections. However, system combination incurs a high computational cost due to the need to run inference on the base systems before running the combination method itself. Therefore, it would be more efficient to have a single model with multiple sub-networks that specialize in correcting different error types. In this paper, we propose a mixture-of-experts model, MoECE, for grammatical error correction. Our model successfully achieves the performance of T5-XL with three times fewer effective parameters. Additionally, our model produces interpretable corrections by also identifying the error type during inference.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 44 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

23 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  6. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
  7. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  8. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  9. 2021A Simple Recipe for Multilingual Grammatical Error Correctioncited by 108
  10. 2020Seq2Edits: Sequence Transduction Using Span-level Edit Operationscited by 58
  11. 2023Enhancing Grammatical Error Correction Systems with Explanationscited by 19
  12. 2020Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analysescited by 16
5 more
  1. 2019Learning to combine Grammatical Error Correctionscited by 9
  2. 2021System Combination for Grammatical Error Correction Based on Integer Programmingcited by 7
  3. 2016Neural Network Translation Models for Grammatical Error Correctioncited by 5
  4. 2023Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedulecited by 5
  5. 2021Diversity-Driven Combination for Grammatical Error Correctioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Leveraging What's Overfixed: Post-Correction via LLM Grammatical Error Overcorrectioncited by 2
  2. 2025JELV: A Judge of Edit-Level Validity for Evaluation and Automated Reference Expansion in Grammatical Error Correctioncited by 0
  3. 2026Incentives Of EdTech: A Systematic Review Of EduNLP Researchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models
  2. 2024NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model
  3. 2024Leveraging LLMs for Bangla Grammar Error Correction:Error Categorization, Synthetic Data, and Model Evaluation
  4. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
  5. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation
  6. 2024ChatLang-8: An LLM-Based Synthetic Data Generation Framework for Grammatical Error Correction
  7. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  8. 2024A Survey on Mixture of Experts in Large Language Models