# Ensemble Distillation Approaches for Grammatical Error Correction

arXiv: https://arxiv.org/abs/2012.07535 · HTML: https://mlsanity.com/paper/2012.07535/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 8 of 8)

- [Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM](https://mlsanity.com/paper/2401.02994/) (2024)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Self-Distribution Distillation: Efficient Uncertainty Estimation](https://mlsanity.com/paper/2203.08295/) (2022)
- [CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models](https://mlsanity.com/paper/2306.05317/) (2023)
- [Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties](https://mlsanity.com/paper/2305.10384/) (2023)
- [Minimum Bayes' Risk Decoding for System Combination of Grammatical Error Correction Systems](https://mlsanity.com/paper/2309.06520/) (2023)
- [Who Needs Decoders? Efficient Estimation of Sequence-level Attributes](https://mlsanity.com/paper/2305.05098/) (2023)
- [Chinese grammatical error correction based on knowledge distillation](https://mlsanity.com/paper/2208.00351/) (2022)

## Abstract (arXiv)

Ensemble approaches are commonly used techniques to improving a system by combining multiple model predictions. Additionally these schemes allow the uncertainty, as well as the source of the uncertainty, to be derived for the prediction. Unfortunately these benefits come at a computational and memory cost. To address this problem ensemble distillation (EnD) and more recently ensemble distribution distillation (EnDD) have been proposed that compress the ensemble into a single model, representing either the ensemble average prediction or prediction distribution respectively. This paper examines the application of both these distillation approaches to a sequence prediction task, grammatical error correction (GEC). This is an important application area for language learning tasks as it can yield highly useful feedback to the learner. It is, however, more challenging than the standard tasks investigated for distillation as the prediction of any grammatical correction to a word will be highly dependent on both the input sequence and the generated output history for the word. The performance of both EnD and EnDD are evaluated on both publicly available GEC tasks as well as a spoken language task.