# What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study

arXiv: https://arxiv.org/abs/2110.04845 · HTML: https://mlsanity.com/paper/2110.04845/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 65 references that are in the map.
- First posted on arXiv 10 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [Fine-Grained Spoiler Detection from Large-Scale Review Corpora](https://mlsanity.com/paper/1905.13416/) (2019)
- [Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling](https://mlsanity.com/paper/1712.01741/) (2017)
- [Practical and Ethical Considerations in the Effective use of Emotion and Sentiment Lexicons](https://mlsanity.com/paper/2011.03492/) (2020)

## Built on by (most cited first, 15 of 15)

- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [Best Practices in the Creation and Use of Emotion Lexicons](https://mlsanity.com/paper/2210.07206/) (2022)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions](https://mlsanity.com/paper/2212.09825/) (2022)
- [Relational Sentence Embedding for Flexible Semantic Matching](https://mlsanity.com/paper/2212.08802/) (2022)
- [4bit-Quantization in Vector-Embedding for RAG](https://mlsanity.com/paper/2501.10534/) (2025)
- [UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation](https://mlsanity.com/paper/2402.12730/) (2024)
- [Banyan: Improved Representation Learning with Explicit Structure](https://mlsanity.com/paper/2407.17771/) (2024)
- [C-STS: Conditional Semantic Textual Similarity](https://mlsanity.com/paper/2305.15093/) (2023)
- [Sharif-STR at SemEval-2024 Task 1: Transformer as a Regression Model for Fine-Grained Scoring of Textual Semantic Relations](https://mlsanity.com/paper/2407.12426/) (2024)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)
- [Anime Popularity Prediction Before Huge Investments: a Multimodal Approach Using Deep Learning](https://mlsanity.com/paper/2406.16961/) (2024)
- [SinaTools: Open Source Toolkit for Arabic Natural Language Processing](https://mlsanity.com/paper/2411.01523/) (2024)
- [Towards Explainable Job Title Matching: Leveraging Semantic Textual Relatedness and Knowledge Graphs](https://mlsanity.com/paper/2509.09522/) (2025)

## Abstract (arXiv)

The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. Additionally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR-2022, that has 5,500 English sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks. Our dataset, data statement, and annotation questionnaire can be found at: https://doi.org/10.5281/zenodo.7599667