From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models
Not on the map. Why?
Abstract (arXiv)
Arabic Language Models (LMs) are pretrained predominately on Modern Standard Arabic (MSA) and are expected to transfer to its dialects. While MSA as the standard written variety is commonly used in formal settings, people speak and write online in various dialects that are spread across the Arab region. This poses limitations for Arabic LMs, since its dialects vary in their similarity to MSA. In this work we study cross-lingual transfer of Arabic models using probing on 3 Natural Language Processing (NLP) Tasks, and representational similarity. Our results indicate that transfer is possible but disproportionate across dialects, which we find to be partially explained by their geographic proximity. Furthermore, we find evidence for negative interference in models trained to support all Arabic dialects. This questions their degree of similarity, and raises concerns for cross-lingual transfer in Arabic models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 66 references that are in the map. First posted on arXiv 10 Feb 2026.
Builds on
32 of 66 references.
References · 32
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
- 2019How multilingual is Multilingual BERT?cited by 642
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
- 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
- 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
- 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
- 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
- 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
20 more
- 2020Information-Theoretic Probing for Linguistic Structurecited by 204
- 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
- 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 164
- 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
- 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
- 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
- 2020AraGPT2: Pre-Trained Transformer for Arabic Language Generationcited by 54
- 2022Wojood: Nested Arabic Named Entity Corpus and Recognition using BERTcited by 32
- 2018NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networkscited by 31
- 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
- 2023ALDi: Quantifying the Arabic Level of Dialectness of Textcited by 24
- 2014LABR: A Large Scale Arabic Sentiment Analysis Benchmarkcited by 22
- 2021DziriBERT: a Pre-trained Language Model for the Algerian Dialectcited by 18
- 2016A Large Scale Corpus of Gulf Arabiccited by 17
- 2023Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Spacecited by 12
- 2024SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corporacited by 7
- 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scriptscited by 7
- 2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6
- 2021Similarities between Arabic Dialects: Investigating Geographical Proximitycited by 5
- 2024EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corporacited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.