mlsanityData to 14 Oct 2026

Domain-specific Pretraining Profile and Transformer Performance: Evidence from Modeling Digital Pragmatics in Arabic-English Code-switching

arXiv 2609.14571 ↗First posted 13 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This study highlights the role of domain-specific pretraining profile (DSPP) in Transformer performance for modeling digital pragmatics in Arabic-English code-switched discourse. It evaluates MARBERT and XLM-R(oBERTa), with BERT serving as a general-purpose baseline. The models were evaluated on their ability to classify context-sensitive pragmatic functions in code-switched social-media discourse. 11695 unique X posts were collected via Python and utilized for the study. The study employs a quantitative and qualitative NLP approach, following a supervised pipeline. Findings unveil that MARBERT consistently surpasses XLM-R with validation Macro F1 increasing from 0.39 to 0.84 and validation loss decreasing from 0.55 to 0.19. On an independent test set, it achieved 0.96 accuracy, 0.83 macro precision, 0.87 macro recall, and 0.85 Macro F1, while XLM-R achieved 0.92 test accuracy but a substantially lower Macro F1 of 0.52. This was also supported by class-level performance where MARBERT outperforms XLM-R considerably with F1 improvements ranging from +0.33 to +0.60, demonstrating a clear advantage in modeling Arabic digital pragmatics. The study concludes that Transformer performance depends more on DSPP than multilingual coverage alone, as the latter does not guarantee optimal performance on a highly specialized pragmatic classification task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 43 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

14 of 43 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  5. 2020AraGPT2: Pre-Trained Transformer for Arabic Language Generationcited by 55
  6. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 41
  7. 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
  8. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  9. 2025SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Datasetcited by 14
  10. 2022Pretrained Domain-Specific Language Model for General Information Retrieval Tasks in the AEC Domaincited by 12
  11. 2024Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasetscited by 5
  12. 2024DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classificationcited by 5
2 more
  1. 2026Machine learning and emoji prediction: How much accuracy can MARBERT achieve?cited by 2
  2. 2026Does Machine "know" interpersonal pragmatics? Evidence from MARBERT's learning of emoji pragmatics in Arabic digital discoursecited by 2

Built on by

Not cited yet.