# From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models

arXiv: https://arxiv.org/abs/2602.09826 · HTML: https://mlsanity.com/paper/2602.09826/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 66 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 30 of 32)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [AraGPT2: Pre-Trained Transformer for Arabic Language Generation](https://mlsanity.com/paper/2012.15520/) (2020)
- [Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT](https://mlsanity.com/paper/2205.09651/) (2022)
- [NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks](https://mlsanity.com/paper/1812.09359/) (2018)
- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [ALDi: Quantifying the Arabic Level of Dialectness of Text](https://mlsanity.com/paper/2310.13747/) (2023)
- [LABR: A Large Scale Arabic Sentiment Analysis Benchmark](https://mlsanity.com/paper/1411.6718/) (2014)
- [DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://mlsanity.com/paper/2109.12346/) (2021)
- [A Large Scale Corpus of Gulf Arabic](https://mlsanity.com/paper/1609.02960/) (2016)
- [Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space](https://mlsanity.com/paper/2305.02151/) (2023)
- [SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://mlsanity.com/paper/2405.06239/) (2024)
- [Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts](https://mlsanity.com/paper/2507.04569/) (2025)
- [Training Bilingual LMs with Data Constraints in the Targeted Language](https://mlsanity.com/paper/2411.12986/) (2024)

## Built on by (most cited first, 1 of 1)

- [Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs](https://mlsanity.com/paper/2607.03936/) (2026)

## Abstract (arXiv)

Arabic Language Models (LMs) are pretrained predominately on Modern Standard Arabic (MSA) and are expected to transfer to its dialects. While MSA as the standard written variety is commonly used in formal settings, people speak and write online in various dialects that are spread across the Arab region. This poses limitations for Arabic LMs, since its dialects vary in their similarity to MSA. In this work we study cross-lingual transfer of Arabic models using probing on 3 Natural Language Processing (NLP) Tasks, and representational similarity. Our results indicate that transfer is possible but disproportionate across dialects, which we find to be partially explained by their geographic proximity. Furthermore, we find evidence for negative interference in models trained to support all Arabic dialects. This questions their degree of similarity, and raises concerns for cross-lingual transfer in Arabic models.