# Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language

arXiv: https://arxiv.org/abs/2602.18964 · HTML: https://mlsanity.com/paper/2602.18964/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 30 references that are in the map.
- First posted on arXiv 21 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://mlsanity.com/paper/2302.08956/) (2023)
- [A Large Self-Annotated Corpus for Sarcasm](https://mlsanity.com/paper/1704.05579/) (2017)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages](https://mlsanity.com/paper/2305.13989/) (2023)
- [SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding](https://mlsanity.com/paper/2408.11319/) (2024)
- [Pragmatic Metacognitive Prompting Improves LLM Performance on Sarcasm Detection](https://mlsanity.com/paper/2412.04509/) (2024)
- [A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection](https://mlsanity.com/paper/1805.11869/) (2018)
- [Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospects](https://mlsanity.com/paper/2502.17364/) (2025)

## Abstract (arXiv)

Sarcasm detection poses a fundamental challenge in computational semantics, requiring models to resolve disparities between literal and intended meaning. The challenge is amplified in low-resource languages where annotated datasets are scarce or nonexistent. We present Yor-Sarc, the first gold-standard dataset for sarcasm detection in Yor`ub'a, a tonal Niger-Congo language spoken by over 50 million people. The dataset comprises 436 instances annotated by three native speakers from diverse dialectal backgrounds using an annotation protocol specifically designed for Yor`ub'a sarcasm by taking culture into account. This protocol incorporates context-sensitive interpretation and community-informed guidelines and is accompanied by a comprehensive analysis of inter-annotator agreement to support replication in other African languages. Substantial to almost perfect agreement was achieved (Fleiss' kappa = 0.7660; pairwise Cohen's kappa = 0.6732--0.8743), with 83.3% unanimous consensus. One annotator pair achieved almost perfect agreement (kappa = 0.8743; 93.8% raw agreement), exceeding a number of reported benchmarks for English sarcasm research works. The remaining 16.7% majority-agreement cases are preserved as soft labels for uncertainty-aware modelling. Yor-Sarcfootnotehttps://github.com/toheebadura/yor-sarc is expected to facilitate research on semantic interpretation and culturally informed NLP for low-resource African languages.