mlsanityData to 14 Oct 2026

ALLaM: Large Language Models for Arabic and English

arXiv 2407.15390 ↗First posted 22 Jul 2024cs.CL cs.AIcited by 55

Not on the map. Why?

Abstract (arXiv)

We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 72 of its 111 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

72 of 111 references.

References · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  13. 2018Deep contextualized word representationscited by 2,290
  14. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  15. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  16. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  17. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  18. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  19. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  20. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  21. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  22. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  23. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  24. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  25. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  26. 2020Aligning AI With Shared Human Valuescited by 723
  27. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  28. 2019How multilingual is Multilingual BERT?cited by 645
  29. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  30. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  31. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  32. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  33. 2017Word Translation Without Parallel Datacited by 487
  34. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  35. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  36. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  37. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  38. 2018Learning Word Vectors for 157 Languagescited by 280

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 29; 2026: 22.

55 papers.

Sort:

By area:

  1. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  2. 2025Commonsense Reasoning in Arab Culturecited by 16
  3. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLMcited by 16
  4. 2024Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialectcited by 13
  5. 2024A Survey of Large Language Models for Arabic Language and its Dialectscited by 13
  6. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  7. 2025Lost in the Mix: Evaluating LLM Understanding of Code-Switched Textcited by 10
  8. 2025Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluationcited by 10
  9. 2025SpokenNativQA: Multilingual Everyday Spoken Queries for LLMscited by 8
  10. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Modelscited by 6
  11. 2025Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scalecited by 5
  12. 2025Large Language Models and Arabic Content: A Reviewcited by 4
38 more
  1. 2025QU-NLP at QIAS 2025 Shared Task: A Two-Phase LLM Fine-Tuning and Retrieval-Augmented Generation Approach for Islamic Inheritance Reasoningcited by 4
  2. 2025Kuwain 1.5B: An Arabic SLM via Language Injectioncited by 3
  3. 2025The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Textcited by 3
  4. 2025Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variantscited by 3
  5. 2025LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoringcited by 3
  6. 2026MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmarkcited by 3
  7. 2026Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Taxcited by 3
  8. 2025Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization Systemcited by 2
  9. 2025Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Modelcited by 2
  10. 2025Are LLMs Good Text Diacritizers? An Arabic and Yoruba Case Studycited by 2
  11. 2025From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Abshercited by 2
  12. 2025Saudi-Dialect-ALLaM: LoRA Fine-Tuning for Dialectal Arabic Generationcited by 2
  13. 2025UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chatcited by 2
  14. 2025AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Modelscited by 2
  15. 2026IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variationcited by 2
  16. 2026SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoningcited by 2
  17. 2026HalluScore: Large Language Model Hallucination Question Answering Benchmarkcited by 2
  18. 2024Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabiccited by 1
  19. 2025Sovereign Large Language Models: Advantages, Strategy and Regulationscited by 1
  20. 2025GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Datacited by 1
  21. 2025Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarkscited by 1
  22. 2025Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Casescited by 1
  23. 2026Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMscited by 1
  24. 2026Grounding Arabic LLMs in the Doha Historical Dictionary: Retrieval-Augmented Understanding of Quran and Hadithcited by 1
  25. 2026Linear Semantic Segmentation for Low-Resource Spoken Dialectscited by 1
  26. 2026WASIL: In-the-Wild Arabic Spoken Interactions with LLMscited by 1
  27. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 1
  28. 2025Sadeed: Advancing Arabic Diacritization Through Small Language Modelcited by 0
  29. 2025Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Taskscited by 0
  30. 2025SmolKalam: Ensemble Quality-Filtered Translation at Scale for High Quality Arabic Post-Training Datacited by 0
  31. 2025Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languagescited by 0
  32. 2026Arabic Prompts with English Tools: A Benchmarkcited by 0
  33. 2026Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approachcited by 0
  34. 2026State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillationcited by 0
  35. 2026Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialoguescited by 0
  36. 2026LLM-Based Financial Sentiment Analysis in Arabic: Evidence from Saudi Marketscited by 0
  37. 2026Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AIcited by 0
  38. 2026Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMscited by 0

50 most cited of 55 All 55 (JSON)