# SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages

arXiv: https://arxiv.org/abs/2402.08638 · HTML: https://mlsanity.com/paper/2402.08638/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 73 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [Spanish Pre-trained BERT Model and Evaluation Data](https://mlsanity.com/paper/2308.02976/) (2023)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [Stance and Sentiment in Tweets](https://mlsanity.com/paper/1605.01655/) (2016)
- [Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity](https://mlsanity.com/paper/2003.04866/) (2020)
- [KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi](https://mlsanity.com/paper/2010.12174/) (2020)
- [Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling](https://mlsanity.com/paper/1712.01741/) (2017)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://mlsanity.com/paper/2109.12346/) (2021)
- [IndoSum: A New Benchmark Dataset for Indonesian Text Summarization](https://mlsanity.com/paper/1810.05334/) (2018)
- [Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasets](https://mlsanity.com/paper/2011.01154/) (2020)
- [V=arta: A Large-Scale Headline-Generation Dataset for Indic Languages](https://mlsanity.com/paper/2305.05858/) (2023)
- [Evaluation Benchmarks for Spanish Sentence Representations](https://mlsanity.com/paper/2204.07571/) (2022)
- [Improving Multilingual Semantic Textual Similarity with Shared Sentence Encoder for Low-resource Languages](https://mlsanity.com/paper/1810.08740/) (2018)
- [hauWE: Hausa Words Embedding for Natural Language Processing](https://mlsanity.com/paper/1911.10708/) (2019)
- [SILT: Efficient transformer training for inter-lingual inference](https://mlsanity.com/paper/2103.09635/) (2021)
- [Mukhyansh: A Headline Generation Dataset for Indic Languages](https://mlsanity.com/paper/2311.17743/) (2023)

## Built on by (most cited first, 16 of 16)

- [SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages](https://mlsanity.com/paper/2403.18933/) (2024)
- [AAdaM at SemEval-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatedness](https://mlsanity.com/paper/2404.01490/) (2024)
- [Building Better: Avoiding Pitfalls in Developing Language Resources when Data is Scarce](https://mlsanity.com/paper/2410.12691/) (2024)
- [NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatedness](https://mlsanity.com/paper/2405.00659/) (2024)
- [Banyan: Improved Representation Learning with Explicit Structure](https://mlsanity.com/paper/2407.17771/) (2024)
- [Sharif-STR at SemEval-2024 Task 1: Transformer as a Regression Model for Fine-Grained Scoring of Textual Semantic Relations](https://mlsanity.com/paper/2407.12426/) (2024)
- [T"ubingen-CL at SemEval-2024 Task 1:Ensemble Learning for Semantic Relatedness Estimation](https://mlsanity.com/paper/2410.10585/) (2024)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)
- [L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages](https://mlsanity.com/paper/2509.02503/) (2025)
- [MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness](https://mlsanity.com/paper/2404.02570/) (2024)
- [IITK at SemEval-2024 Task 1: Contrastive Learning and Autoencoders for Semantic Textual Relatedness in Multilingual Texts](https://mlsanity.com/paper/2404.04513/) (2024)
- [Multilingual Evaluation of Semantic Textual Relatedness](https://mlsanity.com/paper/2404.09047/) (2024)
- [SinaTools: Open Source Toolkit for Arabic Natural Language Processing](https://mlsanity.com/paper/2411.01523/) (2024)
- [USTCCTSU at SemEval-2024 Task 1: Reducing Anisotropy for Cross-lingual Semantic Textual Relatedness Task](https://mlsanity.com/paper/2411.18990/) (2024)
- [Uncertainty-driven Embedding Convolution](https://mlsanity.com/paper/2507.20718/) (2025)
- [KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training](https://mlsanity.com/paper/2608.26941/) (2026)

## Abstract (arXiv)

Exploring and quantifying semantic relatedness is central to representing language and holds significant implications across various NLP tasks. While earlier NLP research primarily focused on semantic similarity, often within the English language context, we instead investigate the broader phenomenon of semantic relatedness. In this paper, we present textitSemRel, a new semantic relatedness dataset collection annotated by native speakers across 13 languages: textitAfrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Spanish, and textitTelugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by a relatively limited availability of NLP resources. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. The scores are obtained using a comparative annotation framework. We describe the data collection and annotation processes, challenges when building the datasets, baseline experiments, and their impact and utility in NLP.