# SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages

arXiv: https://arxiv.org/abs/2403.18933 · HTML: https://mlsanity.com/paper/2403.18933/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 71 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](https://mlsanity.com/paper/2104.06979/) (2021)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling](https://mlsanity.com/paper/1712.01741/) (2017)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024)
- [UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation](https://mlsanity.com/paper/2402.12730/) (2024)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)

## Built on by (most cited first, 7 of 7)

- [AAdaM at SemEval-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatedness](https://mlsanity.com/paper/2404.01490/) (2024)
- [POLAR: A Benchmark for Multilingual, Multicultural, and Multi-Event Online Polarization](https://mlsanity.com/paper/2505.20624/) (2025)
- [Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Less](https://mlsanity.com/paper/2404.01860/) (2024)
- [Banyan: Improved Representation Learning with Explicit Structure](https://mlsanity.com/paper/2407.17771/) (2024)
- [FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation](https://mlsanity.com/paper/2504.17311/) (2025)
- [Multilingual Evaluation of Semantic Textual Relatedness](https://mlsanity.com/paper/2404.09047/) (2024)
- [SinaTools: Open Source Toolkit for Arabic Natural Language Processing](https://mlsanity.com/paper/2411.01523/) (2024)

## Abstract (arXiv)

We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phenomenon of semantic relatedness across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by the relatively limited availability of NLP resources. Each instance in the datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. Participating systems were asked to rank sentence pairs by their closeness in meaning (i.e., their degree of semantic relatedness) in the 14 languages in three main tracks: (a) supervised, (b) unsupervised, and (c) crosslingual. The task attracted 163 participants. We received 70 submissions in total (across all tasks) from 51 different teams, and 38 system description papers. We report on the best-performing systems as well as the most common and the most effective approaches for the three different tracks.