mlsanityData to 7 Oct 2026

RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

arXiv 2605.28827 ↗First posted 10 Apr 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Open Arabic large language models split into two classes: sub-1B multilingual models that treat Arabic as an afterthought (Qwen2.5-0.5B, Falcon-H1-0.5B), and 7B-70B Arabic-specialized models that require a server to run (Jais, AceGPT, ALLaM, SILMA). The one published attempt at a sub-2B Arabic-specialized model, Kuwain-1.5B, never released its weights. We present RightNow-Arabic-0.5B-Turbo, a 518M-parameter Arabic-specialized decoder LLM built on Qwen2.5-0.5B. The pipeline adds 27,032 Arabic tokens via mean-subtoken initialization, continues pretraining on 504M Arabic tokens on 8xH100 with FSDP, FlashAttention varlen packing, and Liger fused kernels, then applies supervised fine-tuning on 129,116 Arabic instruction pairs with response-only loss masking, direct preference optimization on 6,750 Arabic preference pairs, and weight soup merging across three checkpoints. On three lm-evaluation-harness Arabic benchmarks (COPA-ar, Arabic HellaSwag, ArabicMMLU) the merged model reaches 35.9% mean accuracy, beats every same-class open model, ties Falcon-H1-1.5B on COPA-ar (58.4%) at one-third the size, and recovers 67% of SILMA-9B's mean at 1/18 the parameters. The edge build quantizes to 398 MB (q4_k_m) and delivers 635 tokens/s at batch size 1 on a single H100 via llama.cpp. All code (5,555 lines across 25 scripts), weights (bf16, int8, and four GGUF quantizations), and benchmark scripts are released at https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 21 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

17 of 21 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
  12. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
5 more
  1. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  2. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  3. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  4. 2025Kuwain 1.5B: An Arabic SLM via Language Injectioncited by 3
  5. 2025Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Modelcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
  2. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM
  3. 2024CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks
  4. 2025UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat
  5. 2025GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data
  6. 2025Tool Calling for Arabic LLMs: Data Strategies and Instruction Tuning
  7. 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts
  8. 2026AraSSM: A bidirectional state-space encoder for Arabic masked language modeling