# Reference-less Measure of Faithfulness for Grammatical Error Correction

arXiv: https://arxiv.org/abs/1804.03824 · HTML: https://mlsanity.com/paper/1804.03824/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2018.

## Built on by (most cited first, 9 of 9)

- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond](https://mlsanity.com/paper/2205.11484/) (2022)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [ComSum: Commit Messages Summarization and Meaning Preservation](https://mlsanity.com/paper/2108.10763/) (2021)
- [Revisiting Meta-evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2403.02674/) (2024)
- [Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Models](https://mlsanity.com/paper/2505.06004/) (2025)
- [MuLER: Detailed and Scalable Reference-based Evaluation](https://mlsanity.com/paper/2305.14991/) (2023)
- [Don't Count the Edits, Judge by the Outcome Alone: Reward-Based Evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2609.15559/) (2026)

## Abstract (arXiv)

We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USim operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USim, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USim similarity score to the source; and (3) invalid corrections obtain a lower score.