# On the Evaluation Metrics for Paraphrase Generation

arXiv: https://arxiv.org/abs/2202.08479 · HTML: https://mlsanity.com/paper/2202.08479/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [Neural Machine Translation with Monolingual Translation Memory](https://mlsanity.com/paper/2105.11269/) (2021)
- [Neural Syntactic Preordering for Controlled Paraphrase Generation](https://mlsanity.com/paper/2005.02013/) (2020)
- [Syntax-guided Controlled Generation of Paraphrases](https://mlsanity.com/paper/2005.08417/) (2020)
- [Learning to Paraphrase for Question Answering](https://mlsanity.com/paper/1708.06022/) (2017)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [Unsupervised Paraphrasing by Simulated Annealing](https://mlsanity.com/paper/1909.03588/) (2019)
- [ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters](https://mlsanity.com/paper/1809.09528/) (2018)
- [Unsupervised Paraphrasing via Deep Reinforcement Learning](https://mlsanity.com/paper/2007.02244/) (2020)
- [Don't paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing](https://mlsanity.com/paper/1908.09940/) (2019)
- [Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering](https://mlsanity.com/paper/1806.04330/) (2018)
- [A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy](https://mlsanity.com/paper/2106.13945/) (2021)
- [Unsupervised Dual Paraphrasing for Two-stage Semantic Parsing](https://mlsanity.com/paper/2005.13485/) (2020)
- [Automatic Metric Validation for Grammatical Error Correction](https://mlsanity.com/paper/1804.11225/) (2018)
- [Assessing Dialogue Systems with Distribution Distances](https://mlsanity.com/paper/2105.02573/) (2021)
- [Unsupervised Paraphrase Generation using Pre-trained Language Models](https://mlsanity.com/paper/2006.05477/) (2020)
- [Unsupervised Paraphrasing with Pretrained Language Models](https://mlsanity.com/paper/2010.12885/) (2020)
- [From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decoding](https://mlsanity.com/paper/2106.06228/) (2021)
- [NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation](https://mlsanity.com/paper/1810.00671/) (2018)
- [REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation](https://mlsanity.com/paper/2105.14488/) (2021)

## Built on by (most cited first, 30 of 32)

- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Fusing Models with Complementary Expertise](https://mlsanity.com/paper/2310.01542/) (2023)
- [The Trickle-down Impact of Reward (In-)consistency on RLHF](https://mlsanity.com/paper/2309.16155/) (2023)
- [Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability](https://mlsanity.com/paper/2406.18365/) (2024)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [Building Trustworthy NeuroSymbolic AI Systems: Consistency, Reliability, Explainability, and Safety](https://mlsanity.com/paper/2312.06798/) (2023)
- [DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation](https://mlsanity.com/paper/2405.15329/) (2024)
- [LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training](https://mlsanity.com/paper/2401.04348/) (2024)
- [AdParaphrase: Paraphrase Dataset for Analyzing Linguistic Features toward Generating Attractive Ad Texts](https://mlsanity.com/paper/2502.04674/) (2025)
- [Spivavtor: An Instruction Tuned Ukrainian Text Editing Model](https://mlsanity.com/paper/2404.18880/) (2024)
- [Do Audio-Language Models Understand Linguistic Variations?](https://mlsanity.com/paper/2410.16505/) (2024)
- [Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification](https://mlsanity.com/paper/2507.15557/) (2025)
- [Measuring Moral Inconsistencies in Large Language Models](https://mlsanity.com/paper/2402.01719/) (2024)
- [SaGE: Evaluating Moral Consistency in Large Language Models](https://mlsanity.com/paper/2402.13709/) (2024)
- [PlagBench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection](https://mlsanity.com/paper/2406.16288/) (2024)
- [The Ultimate Cookbook for Invisible Poison: Crafting Subtle Clean-Label Text Backdoors with Style Attributes](https://mlsanity.com/paper/2504.17300/) (2025)
- [Dense Paraphrasing for Textual Enrichment](https://mlsanity.com/paper/2210.11563/) (2022)
- [Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Model](https://mlsanity.com/paper/2306.02247/) (2023)
- [A Quality-based Syntactic Template Retriever for Syntactically-controlled Paraphrase Generation](https://mlsanity.com/paper/2310.13262/) (2023)
- [Improved Paraphrase Generation via Controllable Latent Diffusion](https://mlsanity.com/paper/2404.08938/) (2024)
- [Linguistically-Controlled Paraphrase Generation](https://mlsanity.com/paper/2410.24199/) (2024)
- [Context Selection and Rewriting for Video-based Educational Question Generation](https://mlsanity.com/paper/2504.19406/) (2025)
- [Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps](https://mlsanity.com/paper/2505.18426/) (2025)
- [Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation](https://mlsanity.com/paper/2511.16478/) (2025)
- [IdEALS: Idiomatic Expressions for Advancement of Language Skills](https://mlsanity.com/paper/2305.13637/) (2023)
- [A Benchmark for Text Expansion: Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2309.09198/) (2023)
- [AMPS: ASR with Multimodal Paraphrase Supervision](https://mlsanity.com/paper/2411.18368/) (2024)
- [Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data](https://mlsanity.com/paper/2506.02018/) (2025)
- [Helping the Helper: LLM-Assisted Problem Articulation for Older Adults Seeking Technology Support](https://mlsanity.com/paper/2601.10018/) (2026)

## Abstract (arXiv)

In this paper we revisit automatic metrics for paraphrase evaluation and obtain two findings that disobey conventional wisdom: (1) Reference-free metrics achieve better performance than their reference-based counterparts. (2) Most commonly used metrics do not align well with human annotation. Underlying reasons behind the above findings are explored through additional experiments and in-depth analyses. Based on the experiments and analyses, we propose ParaScore, a new evaluation metric for paraphrase generation. It possesses the merits of reference-based and reference-free metrics and explicitly models lexical divergence. Experimental results demonstrate that ParaScore significantly outperforms existing metrics.