mlsanityData to 7 Oct 2026

Zero-Resource Multi-Dialectal Arabic Natural Language Understanding

arXiv 2104.06591 ↗First posted 14 Apr 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A reasonable amount of annotated data is required for fine-tuning pre-trained language models (PLM) on downstream tasks. However, obtaining labeled examples for different language varieties can be costly. In this paper, we investigate the zero-shot performance on Dialectal Arabic (DA) when fine-tuning a PLM on modern standard Arabic (MSA) data only -- identifying a significant performance drop when evaluating such models on DA. To remedy such performance drop, we propose self-training with unlabeled DA data and apply it in the context of named entity recognition (NER), part-of-speech (POS) tagging, and sarcasm detection (SRD) on several DA varieties. Our results demonstrate the effectiveness of self-training with unlabeled DA data: improving zero-shot MSA-to-DA transfer by as large as ~10% F1 (NER), 2% accuracy (POS tagging), and 4.5% F1 (SRD). We conduct an ablation experiment and show that the performance boost observed directly results from the unlabeled DA examples used for self-training. Our work opens up opportunities for leveraging the relatively abundant labeled MSA datasets to develop DA models for zero and low-resource dialects. We also report new state-of-the-art performance on all three tasks and open-source our fine-tuned models for the research community.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 112 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

24 of 112 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
  10. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  12. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
12 more
  1. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  2. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  3. 2016Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classificationcited by 48
  4. 2018Neural Cross-Lingual Named Entity Recognition with Minimal Resourcescited by 44
  5. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  6. 2019A Transformer-based approach to Irony and Sarcasm detectioncited by 28
  7. 2019Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NERcited by 25
  8. 2016Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoderscited by 13
  9. 2020Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classificationcited by 11
  10. 2016Predicting the Effectiveness of Self-Training: Application to Sentiment Classificationcited by 5
  11. 2019A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Wellcited by 4
  12. 2019Multi-Task Bidirectional Transformer Representations for Irony Detectioncited by 2

Built on by

Not cited yet.