mlsanityData to 7 Oct 2026

AlcLaM: Arabic Dialectal Language Model

arXiv 2407.13097 ↗First posted 18 Jul 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models (PLMs) are integral to many modern natural language processing (NLP) systems. Although multilingual models cover a wide range of languages, they often grapple with challenges like high inference costs and a lack of diverse non-English training data. Arabic-specific PLMs are trained predominantly on modern standard Arabic, which compromises their performance on regional dialects. To tackle this, we construct an Arabic dialectal corpus comprising 3.4M sentences gathered from social media platforms. We utilize this corpus to expand the vocabulary and retrain a BERT-based model from scratch. Named AlcLaM, our model was trained using only 13 GB of text, which represents a fraction of the data used by existing models such as CAMeL, MARBERT, and ArBERT, compared to 7.8%, 10.2%, and 21.3%, respectively. Remarkably, AlcLaM demonstrates superior performance on a variety of Arabic NLP tasks despite the limited training data. AlcLaM is available at GitHub https://github.com/amurtadha/Alclam and HuggingFace https://huggingface.co/rahbi.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

30 of 41 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  12. 2020Towards a Human-like Open-Domain Chatbotcited by 467
18 more
  1. 2019CamemBERT: a Tasty French Language Modelcited by 368
  2. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  3. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 164
  4. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  5. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
  6. 2019Multilingual is not enough: BERT for Finnishcited by 118
  7. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  8. 2019BERTje: A Dutch BERT Modelcited by 109
  9. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  10. 2020NADI 2020: The First Nuanced Arabic Dialect Identification Shared Taskcited by 47
  11. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  12. 2014LABR: A Large Scale Arabic Sentiment Analysis Benchmarkcited by 22
  13. 2020Pre-training Polish Transformer-based Language Models at Scalecited by 21
  14. 2020WikiBERT models: deep transfer learning for many languagescited by 20
  15. 2020Multi-Dialect Arabic BERT for Country-Level Dialect Identificationcited by 15
  16. 2020ASAD: A Twitter-based Benchmark Arabic Sentiment Analysis Datasetcited by 7
  17. 2022BERT-ASC: Auxiliary-Sentence Construction for Implicit Aspect Learning in Sentiment Analysiscited by 4
  18. 2023Rank-Aware Negative Training for Semi-Supervised Text Classificationcited by 3

Built on by

Not cited yet.