# Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric

arXiv: https://arxiv.org/abs/2004.05001 · HTML: https://mlsanity.com/paper/2004.05001/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [XFORMAL: A Benchmark for Multilingual Formality Style Transfer](https://mlsanity.com/paper/2104.04108/) (2021)
- [Text Detoxification using Large Pre-trained Neural Models](https://mlsanity.com/paper/2109.08914/) (2021)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [Methods for Detoxification of Texts for the Russian Language](https://mlsanity.com/paper/2105.09052/) (2021)
- [Text Style Transfer Evaluation Using Large Language Models](https://mlsanity.com/paper/2308.13577/) (2023)
- [Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer](https://mlsanity.com/paper/2204.07549/) (2022)
- [Multidimensional Evaluation for Text Style Transfer Using ChatGPT](https://mlsanity.com/paper/2304.13462/) (2023)
- [Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?](https://mlsanity.com/paper/2502.04718/) (2025)
- [T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation](https://mlsanity.com/paper/2212.01667/) (2022)
- [StyleKQC: A Style-Variant Paraphrase Corpus for Korean Questions and Commands](https://mlsanity.com/paper/2103.13439/) (2021)
- [What is Wrong with Language Models that Can Not Tell a Story?](https://mlsanity.com/paper/2211.05044/) (2022)
- [Rethinking Crowd Sourcing for Semantic Similarity](https://mlsanity.com/paper/2109.11969/) (2021)
- [Utilizing Semantic Textual Similarity for Clinical Survey Data Feature Selection](https://mlsanity.com/paper/2308.09892/) (2023)
- [EENLP: Cross-lingual Eastern European NLP Index](https://mlsanity.com/paper/2108.02605/) (2021)
- [Gradient-guided Unsupervised Text Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2202.00469/) (2022)
- [Studying the role of named entities for content preservation in text style transfer](https://mlsanity.com/paper/2206.09676/) (2022)
- [Characterizing and Measuring Linguistic Dataset Drift](https://mlsanity.com/paper/2305.17127/) (2023)
- [Moving Other Way: Exploring Word Mover Distance Extensions](https://mlsanity.com/paper/2202.03119/) (2022)
- [Variational Transformer: A Framework Beyond the Trade-off between Accuracy and Diversity for Image Captioning](https://mlsanity.com/paper/2205.14458/) (2022)
- [Neural Machine Translation for Malayalam Paraphrase Generation](https://mlsanity.com/paper/2401.17827/) (2024)
- [Can We Reliably Rank Model Performance across Domains without Labeled Data?](https://mlsanity.com/paper/2510.09519/) (2025)

## Abstract (arXiv)

The rapid development of such natural language processing tasks as style transfer, paraphrase, and machine translation often calls for the use of semantic similarity metrics. In recent years a lot of methods to measure the semantic similarity of two short texts were developed. This paper provides a comprehensive analysis for more than a dozen of such methods. Using a new dataset of fourteen thousand sentence pairs human-labeled according to their semantic similarity, we demonstrate that none of the metrics widely used in the literature is close enough to human judgment in these tasks. A number of recently proposed metrics provide comparable results, yet Word Mover Distance is shown to be the most reasonable solution to measure semantic similarity in reformulated texts at the moment.