# AceGPT, Localizing Large Language Models in Arabic

arXiv: https://arxiv.org/abs/2309.12053 · HTML: https://mlsanity.com/paper/2309.12053/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering](https://mlsanity.com/paper/2011.03080/) (2020)
- [Knowledge of cultural moral norms in large language models](https://mlsanity.com/paper/2306.01857/) (2023)
- [Phoenix: Democratizing ChatGPT across Languages](https://mlsanity.com/paper/2304.10453/) (2023)
- [Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailment](https://mlsanity.com/paper/2305.14492/) (2023)
- [AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing](https://mlsanity.com/paper/2306.06800/) (2023)

## Built on by (most cited first, 30 of 75)

- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [Self-Pluralising Culture Alignment for Large Language Models](https://mlsanity.com/paper/2410.12971/) (2024)
- [NativQA: Multilingual Culturally-Aligned Natural Query for LLMs](https://mlsanity.com/paper/2407.09823/) (2024)
- [Multiple LLM Agents Debate for Equitable Cultural Alignment](https://mlsanity.com/paper/2505.24671/) (2025)
- [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://mlsanity.com/paper/2509.02550/) (2025)
- [CIDAR: Culturally Relevant Instruction Dataset For Arabic](https://mlsanity.com/paper/2402.03177/) (2024)
- [Arabic Dataset for LLM Safeguard Evaluation](https://mlsanity.com/paper/2410.17040/) (2024)
- [AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic](https://mlsanity.com/paper/2403.09017/) (2024)
- [Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic](https://mlsanity.com/paper/2407.18129/) (2024)
- [Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette](https://mlsanity.com/paper/2412.11167/) (2024)
- [Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM](https://mlsanity.com/paper/2503.14603/) (2025)
- [Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs](https://mlsanity.com/paper/2505.18152/) (2025)
- [Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs](https://mlsanity.com/paper/2503.00151/) (2025)
- [Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect](https://mlsanity.com/paper/2409.17912/) (2024)
- [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024)
- [NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities](https://mlsanity.com/paper/2505.18383/) (2025)
- [The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology](https://mlsanity.com/paper/2506.01340/) (2025)
- [AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic](https://mlsanity.com/paper/2412.04193/) (2024)
- [Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation](https://mlsanity.com/paper/2509.01081/) (2025)
- [MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language](https://mlsanity.com/paper/2409.12257/) (2024)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian](https://mlsanity.com/paper/2502.07459/) (2025)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [ArabianGPT: Native Arabic GPT-based Large Language Model](https://mlsanity.com/paper/2402.15313/) (2024)
- [Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts](https://mlsanity.com/paper/2507.04569/) (2025)
- [IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge](https://mlsanity.com/paper/2603.23750/) (2026)
- [How well can LLMs Grade Essays in Arabic?](https://mlsanity.com/paper/2501.16516/) (2025)
- [LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?](https://mlsanity.com/paper/2503.15003/) (2025)

## Abstract (arXiv)

This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) employing a reward model attuned to local culture and values. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities. Comprehensive evaluations reveal that the resulting model, dubbed `AceGPT', sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. Codes, data, and models are in https://github.com/FreedomIntelligence/AceGPT.