# Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really Need Reference?

arXiv: https://arxiv.org/abs/2312.01500 · HTML: https://mlsanity.com/paper/2312.01500/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://mlsanity.com/paper/1710.02187/) (2017)
- [Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!](https://mlsanity.com/paper/1809.08731/) (2018)
- [Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT](https://mlsanity.com/paper/1802.06041/) (2018)

## Built on by (most cited first, 2 of 2)

- [ConfusionPrompt: Practical Private Inference for Online Large Language Models](https://mlsanity.com/paper/2401.00870/) (2023)
- [ContextBench: Modifying Contexts for Targeted Latent Activation](https://mlsanity.com/paper/2506.15735/) (2025)

## Abstract (arXiv)

Fluency is a crucial goal of all Natural Language Generation (NLG) systems. Widely used automatic evaluation metrics fall short in capturing the fluency of machine-generated text. Assessing the fluency of NLG systems poses a challenge since these models are not limited to simply reusing words from the input but may also generate abstractions. Existing reference-based fluency evaluations, such as word overlap measures, often exhibit weak correlations with human judgments. This paper adapts an existing unsupervised technique for measuring text fluency without the need for any reference. Our approach leverages various word embeddings and trains language models using Recurrent Neural Network (RNN) architectures. We also experiment with other available multilingual Language Models (LMs). To assess the performance of the models, we conduct a comparative analysis across 10 Indic languages, correlating the obtained fluency scores with human judgments. Our code and human-annotated benchmark test-set for fluency is available at https://github.com/AnanyaCoder/TextFluencyForIndicLanaguges.