mlsanityData to 14 Oct 2026

Collective Human Opinions in Semantic Textual Similarity

arXiv 2308.04114 ↗First posted 8 Aug 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Despite the subjective nature of semantic textual similarity (STS) and pervasive disagreements in STS annotation, existing benchmarks have used averaged human ratings as the gold standard. Averaging masks the true distribution of human opinions on examples of low agreement, and prevents models from capturing the semantic vagueness that the individual ratings represent. In this work, we introduce USTS, the first Uncertainty-aware STS dataset with ~15,000 Chinese sentence pairs and 150,000 labels, to study collective human opinions in STS. Analysis reveals that neither a scalar nor a single Gaussian fits a set of observed judgements adequately. We further show that current STS models cannot capture the variance caused by human disagreement on individual instances, but rather reflect the predictive confidence over the aggregate dataset.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

20 of 47 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  11. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  12. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
8 more
  1. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  2. 2021Generating Datasets with Pretrained Language Modelscited by 193
  3. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  4. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  5. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  6. 2019Uncertain Natural Language Inferencecited by 35
  7. 2021Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linkingcited by 28
  8. 2021Robustness and Reliability When Training With Noisy Labelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 5.

8 papers.

  1. 2023Human Feedback is not Gold Standardcited by 83
  2. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 31
  3. 2026STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systemscited by 3
  4. 2023Rethinking STS and NLI in Large Language Modelscited by 2
  5. 2026Quantifying and Predicting Disagreement in Graded Human Ratingscited by 1
  6. 2026The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variationcited by 0
  7. 2026Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probingcited by 0
  8. 2026Chinese-Jev: Bringing System One Model to Chinese-Language Taskscited by 0