# Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection

arXiv: https://arxiv.org/abs/1904.07334 · HTML: https://mlsanity.com/paper/1904.07334/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2019.

## Built on by (most cited first, 7 of 7)

- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://mlsanity.com/paper/2305.14734/) (2023)
- [Interpretability for Language Learners Using Example-Based Grammatical Error Correction](https://mlsanity.com/paper/2203.07085/) (2022)
- [Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction](https://mlsanity.com/paper/2105.04443/) (2021)
- [Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors](https://mlsanity.com/paper/2108.12216/) (2021)
- [Oddballness: universal anomaly detection with language models](https://mlsanity.com/paper/2409.03046/) (2024)
- [Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detection](https://mlsanity.com/paper/2407.11854/) (2024)

## Abstract (arXiv)

It is known that a deep neural network model pre-trained with large-scale data greatly improves the accuracy of various tasks, especially when there are resource constraints. However, the information needed to solve a given task can vary, and simply using the output of the final layer is not necessarily sufficient. Moreover, to our knowledge, exploiting large language representation models to detect grammatical errors has not yet been studied. In this work, we investigate the effect of utilizing information not only from the final layer but also from intermediate layers of a pre-trained language representation model to detect grammatical errors. We propose a multi-head multi-layer attention model that determines the appropriate layers in Bidirectional Encoder Representation from Transformers (BERT). The proposed method achieved the best scores on three datasets for grammatical error detection tasks, outperforming the current state-of-the-art method by 6.0 points on FCE, 8.2 points on CoNLL14, and 12.2 points on JFLEG in terms of F_0.5. We also demonstrate that by using multi-head multi-layer attention, our model can exploit a broader range of information for each token in a sentence than a model that uses only the final layer's information.