mlsanityData to 14 Oct 2026

JABER and SABER: Junior and Senior Arabic BERt

arXiv 2112.04329 ↗First posted 8 Dec 2021cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception. However, we found that previously released Arabic BERT models were significantly under-trained. In this technical report, we present JABER and SABER, Junior and Senior Arabic BERt respectively, our pre-trained language model prototypes dedicated for Arabic. We conduct an empirical study to systematically evaluate the performance of models across a diverse set of existing Arabic NLU tasks. Experimental results show that JABER and SABER achieve state-of-the-art performances on ALUE, a new benchmark for Arabic Language Understanding Evaluation, as well as on a well-established NER benchmark.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

35 of 57 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
23 more
  1. 2019How multilingual is Multilingual BERT?cited by 645
  2. 2019Green AIcited by 627
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  4. 2019CamemBERT: a Tasty French Language Modelcited by 368
  5. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  6. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  7. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  8. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  9. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  10. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  11. 2018A Simple Method for Commonsense Reasoningcited by 223
  12. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  13. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  14. 2020German's Next Language Modelcited by 144
  15. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  16. 2019Multilingual is not enough: BERT for Finnishcited by 119
  17. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  18. 2019BERTje: A Dutch BERT Modelcited by 109
  19. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  20. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  21. 2019NEZHA: Neural Contextualized Representation for Chinese Language Understandingcited by 39
  22. 2020FinEst BERT and CroSloEngual BERT: less is more in multilingual modelscited by 20
  23. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  2. 2024On the importance of Data Scale in Pretraining Arabic Language Modelscited by 0
  3. 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMscited by 0