# Zero-Resource Multi-Dialectal Arabic Natural Language Understanding

arXiv: https://arxiv.org/abs/2104.06591 · HTML: https://mlsanity.com/paper/2104.06591/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 112 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification](https://mlsanity.com/paper/1606.01614/) (2016)
- [Neural Cross-Lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1808.09861/) (2018)
- [Small and Practical BERT Models for Sequence Labeling](https://mlsanity.com/paper/1909.00100/) (2019)
- [A Transformer-based approach to Irony and Sarcasm detection](https://mlsanity.com/paper/1911.10401/) (2019)
- [Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER](https://mlsanity.com/paper/1909.00153/) (2019)
- [Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders](https://mlsanity.com/paper/1608.02996/) (2016)
- [Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification](https://mlsanity.com/paper/2007.15072/) (2020)
- [Predicting the Effectiveness of Self-Training: Application to Sentiment Classification](https://mlsanity.com/paper/1601.03288/) (2016)
- [A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Well](https://mlsanity.com/paper/1912.01706/) (2019)
- [Multi-Task Bidirectional Transformer Representations for Irony Detection](https://mlsanity.com/paper/1909.03526/) (2019)

## Abstract (arXiv)

A reasonable amount of annotated data is required for fine-tuning pre-trained language models (PLM) on downstream tasks. However, obtaining labeled examples for different language varieties can be costly. In this paper, we investigate the zero-shot performance on Dialectal Arabic (DA) when fine-tuning a PLM on modern standard Arabic (MSA) data only -- identifying a significant performance drop when evaluating such models on DA. To remedy such performance drop, we propose self-training with unlabeled DA data and apply it in the context of named entity recognition (NER), part-of-speech (POS) tagging, and sarcasm detection (SRD) on several DA varieties. Our results demonstrate the effectiveness of self-training with unlabeled DA data: improving zero-shot MSA-to-DA transfer by as large as ~10% F1 (NER), 2% accuracy (POS tagging), and 4.5% F1 (SRD). We conduct an ablation experiment and show that the performance boost observed directly results from the unlabeled DA examples used for self-training. Our work opens up opportunities for leveraging the relatively abundant labeled MSA datasets to develop DA models for zero and low-resource dialects. We also report new state-of-the-art performance on all three tasks and open-source our fine-tuned models for the research community.