# Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets

arXiv: https://arxiv.org/abs/2405.11282 · HTML: https://mlsanity.com/paper/2405.11282/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 18 May 2024.

## Builds on (most cited first, 10 of 10)

- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Overview for the Second Shared Task on Language Identification in Code-Switched Data](https://mlsanity.com/paper/1909.13016/) (2019)
- [NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2310.16117/) (2023)
- [ALDi: Quantifying the Arabic Level of Dialectness of Text](https://mlsanity.com/paper/2310.13747/) (2023)
- [Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://mlsanity.com/paper/2110.06744/) (2021)
- [Towards Responsible Natural Language Annotation for the Varieties of Arabic](https://mlsanity.com/paper/2203.09597/) (2022)
- [Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification](https://mlsanity.com/paper/2310.13661/) (2023)
- [Masader Plus: A New Interface for Exploring +500 Arabic NLP Datasets](https://mlsanity.com/paper/2208.00932/) (2022)

## Built on by (most cited first, 4 of 4)

- [LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?](https://mlsanity.com/paper/2503.15003/) (2025)
- [Revisiting Common Assumptions about Arabic Dialects in NLP](https://mlsanity.com/paper/2505.21816/) (2025)
- [Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages](https://mlsanity.com/paper/2607.02235/) (2026)
- [Voice Conversion Improves Cross-Domain Robustness for Spoken Arabic Dialect Identification](https://mlsanity.com/paper/2505.24713/) (2025)

## Abstract (arXiv)

On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respective dialects to build higher-quality datasets. However, automatically identifying the dialect of samples is hard. Moreover, the pool of annotators who are native speakers of specific Arabic dialects might be scarce. Arabic Level of Dialectness (ALDi) was recently introduced as a quantitative variable that measures how sentences diverge from Standard Arabic. On randomly assigning samples to annotators, we hypothesize that samples of higher ALDi scores are harder to label especially if they are written in dialects that the annotators do not speak. We test this by analyzing the relation between ALDi scores and the annotators' agreement, on 15 public datasets having raw individual sample annotations for various sentence-classification tasks. We find strong evidence supporting our hypothesis for 11 of them. Consequently, we recommend prioritizing routing samples of high ALDi scores to native speakers of each sample's dialect, for which the dialect could be automatically identified at higher accuracies.