# Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!

arXiv: https://arxiv.org/abs/1809.08731 · HTML: https://mlsanity.com/paper/1809.08731/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2018.

## Built on by (most cited first, 30 of 43)

- [Understanding Iterative Revision from Human-Written Text](https://mlsanity.com/paper/2203.03802/) (2022)
- [Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text](https://mlsanity.com/paper/2107.03444/) (2021)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [EditEval: An Instruction-Based Benchmark for Text Improvements](https://mlsanity.com/paper/2209.13331/) (2022)
- [Phrase-level Textual Adversarial Attack with Label Preservation](https://mlsanity.com/paper/2205.10710/) (2022)
- [Towards Imperceptible Document Manipulations against Neural Ranking Models](https://mlsanity.com/paper/2305.01860/) (2023)
- [AGIR: Automating Cyber Threat Intelligence Reporting with Natural Language Generation](https://mlsanity.com/paper/2310.02655/) (2023)
- [TASA: Deceiving Question Answering Models by Twin Answer Sentences Attack](https://mlsanity.com/paper/2210.15221/) (2022)
- [Computational Language Acquisition with Theory of Mind](https://mlsanity.com/paper/2303.01502/) (2023)
- [NAREOR: The Narrative Reordering Problem](https://mlsanity.com/paper/2104.06669/) (2021)
- [How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments](https://mlsanity.com/paper/2408.09639/) (2024)
- [Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization](https://mlsanity.com/paper/2312.10610/) (2023)
- [What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length](https://mlsanity.com/paper/2411.02528/) (2024)
- [A Survey of Data Synthesis Approaches](https://mlsanity.com/paper/2407.03672/) (2024)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [Meta-evaluation of Conversational Search Evaluation Metrics](https://mlsanity.com/paper/2104.13453/) (2021)
- [ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detection](https://mlsanity.com/paper/2402.11167/) (2024)
- [Is Reference Necessary in the Evaluation of NLG Systems? When and Where?](https://mlsanity.com/paper/2403.14275/) (2024)
- [RepEval: Effective Text Evaluation with LLM Representation](https://mlsanity.com/paper/2404.19563/) (2024)
- [Large Language Models Are Active Critics in NLG Evaluation](https://mlsanity.com/paper/2410.10724/) (2024)
- [Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models](https://mlsanity.com/paper/2410.22517/) (2024)
- [Improving Zero-Shot Event Extraction via Sentence Simplification](https://mlsanity.com/paper/2204.02531/) (2022)
- [Rethinking Textual Adversarial Defense for Pre-trained Language Models](https://mlsanity.com/paper/2208.10251/) (2022)
- [Controllable Citation Sentence Generation with Language Models](https://mlsanity.com/paper/2211.07066/) (2022)
- [Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation](https://mlsanity.com/paper/2305.17819/) (2023)
- [Controllable Generation of Dialogue Acts for Dialogue Systems via Few-Shot Response Generation and Ranking](https://mlsanity.com/paper/2307.14440/) (2023)
- [Graphmax for Text Generation](https://mlsanity.com/paper/2101.00153/) (2021)
- [On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation](https://mlsanity.com/paper/2110.03067/) (2021)
- [System Combination via Quality Estimation for Grammatical Error Correction](https://mlsanity.com/paper/2310.14947/) (2023)
- [Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really Need Reference?](https://mlsanity.com/paper/2312.01500/) (2023)

## Abstract (arXiv)

Motivated by recent findings on the probabilistic modeling of acceptability judgments, we propose syntactic log-odds ratio (SLOR), a normalized language model score, as a metric for referenceless fluency evaluation of natural language generation output at the sentence level. We further introduce WPSLOR, a novel WordPiece-based version, which harnesses a more compact language model. Even though word-overlap metrics like ROUGE are computed with the help of hand-written references, our referenceless methods obtain a significantly higher correlation with human fluency scores on a benchmark dataset of compressed sentences. Finally, we present ROUGE-LM, a reference-based metric which is a natural extension of WPSLOR to the case of available references. We show that ROUGE-LM yields a significantly higher correlation with human judgments than all baseline metrics, including WPSLOR on its own.