mlsanityData to 7 Oct 2026

AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

arXiv 2608.08256 ↗First posted 8 Aug 2026cs.CLcited by 0

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Pretrained Transformer encoders such as AraBERT, MARBERT, and CAMeLBERT have become the standard backbone for Arabic natural language understanding, but their self-attention mechanism scales quadratically with sequence length, which limits efficiency on long documents. Mamba, a selective state-space model (SSM), offers linear-time sequence modeling as a competitive alternative to attention, yet no dedicated bidirectional Mamba encoder pretrained specifically for Arabic currently exists. We introduce AraSSM, a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days. We evaluate AraSSM by fine-tuning on four established Arabic NLU benchmarks covering sentiment classification (HARD), named entity recognition (ANERcorp), extractive question answering (ARCD), and natural language inference (XNLI-ar), following the per-task evaluation protocol introduced by AraBERT, and report results as mean +/- standard deviation across three fine-tuning seeds. AraSSM matches or exceeds published base-sized Transformer baselines on sentiment classification (96.37 +/- 0.03% accuracy on HARD), is competitive on extractive QA (32.19 +/- 1.07 EM, 63.79 +/- 0.25 F1 on ARCD) and named entity recognition (81.54 +/- 0.30 entity-level F1 on ANERcorp), and trails the base-sized Transformer range on natural language inference (72.83 +/- 0.07% accuracy on XNLI-ar), despite being trained entirely from scratch on consumer hardware rather than large-scale accelerator clusters.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 17 of its 23 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

17 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  2. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  3. 2024VMamba: Visual State Space Modelcited by 955
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  6. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  7. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  8. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 164
  9. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  10. 2019Neural Arabic Question Answeringcited by 48
  11. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  12. 2020AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understandingcited by 43
3 more
  1. 2014POLYGLOT-NER: Massive Multilingual Named Entity Recognitioncited by 32
  2. 2024ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notescited by 22
  3. 2024CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Datasetcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment
  2. 2024CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks
  3. 2025AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP
  4. 2021Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling
  5. 2025UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat
  6. 2024An Empirical Study of Mamba-based Language Models
  7. 2026State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
  8. 2025Optimizing RAG Pipelines for Arabic: A Systematic Analysis of Core Components