# Rethinking Crowd Sourcing for Semantic Similarity

arXiv: https://arxiv.org/abs/2109.11969 · HTML: https://mlsanity.com/paper/2109.11969/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 30 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric](https://mlsanity.com/paper/2004.05001/) (2020)
- [Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites](https://mlsanity.com/paper/1908.06809/) (2019)
- [Structured Content Preservation for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1810.06526/) (2018)
- [What is wrong with style transfer for texts?](https://mlsanity.com/paper/1808.04365/) (2018)
- [Don't Blame Distributional Semantics if it can't do Entailment](https://mlsanity.com/paper/1905.07356/) (2019)

## Built on by (most cited first, 2 of 2)

- [What is Wrong with Language Models that Can Not Tell a Story?](https://mlsanity.com/paper/2211.05044/) (2022)
- [Moving Other Way: Exploring Word Mover Distance Extensions](https://mlsanity.com/paper/2202.03119/) (2022)

## Abstract (arXiv)

Estimation of semantic similarity is crucial for a variety of natural language processing (NLP) tasks. In the absence of a general theory of semantic information, many papers rely on human annotators as the source of ground truth for semantic similarity estimation. This paper investigates the ambiguities inherent in crowd-sourced semantic labeling. It shows that annotators that treat semantic similarity as a binary category (two sentences are either similar or not similar and there is no middle ground) play the most important role in the labeling. The paper offers heuristics to filter out unreliable annotators and stimulates further discussions on human perception of semantic similarity.