# AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization

arXiv: https://arxiv.org/abs/2203.10945 · HTML: https://mlsanity.com/paper/2203.10945/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://mlsanity.com/paper/1802.08636/) (2018)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection](https://mlsanity.com/paper/2104.09061/) (2021)
- [LCSTS: A Large Scale Chinese Short Text Summarization Dataset](https://mlsanity.com/paper/1506.05865/) (2015)
- [Neural Document Summarization by Jointly Learning to Score and Select Sentences](https://mlsanity.com/paper/1807.02305/) (2018)
- [BARThez: a Skilled Pretrained French Sequence-to-Sequence Model](https://mlsanity.com/paper/2010.12321/) (2020)
- [Pretrained Language Model Embryology: The Birth of ALBERT](https://mlsanity.com/paper/2010.02480/) (2020)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [BERT Fine-tuning For Arabic Text Summarization](https://mlsanity.com/paper/2004.14135/) (2020)

## Built on by (most cited first, 24 of 24)

- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2310.16117/) (2023)
- [NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2407.04910/) (2024)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://mlsanity.com/paper/2305.14734/) (2023)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [BiMediX: Bilingual Medical Mixture of Experts LLM](https://mlsanity.com/paper/2402.13253/) (2024)
- [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024)
- [Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction](https://mlsanity.com/paper/2312.08400/) (2023)
- [GreekBART: The First Pretrained Greek Sequence-to-Sequence Model](https://mlsanity.com/paper/2304.00869/) (2023)
- [AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing](https://mlsanity.com/paper/2306.06800/) (2023)
- [Exploiting Dialect Identification in Automatic Dialectal Text Normalization](https://mlsanity.com/paper/2407.03020/) (2024)
- [Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation](https://mlsanity.com/paper/2310.16127/) (2023)
- [ChatGPT for Arabic Grammatical Error Correction](https://mlsanity.com/paper/2308.04492/) (2023)
- [Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis](https://mlsanity.com/paper/2405.14277/) (2024)
- [EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora](https://mlsanity.com/paper/2408.03524/) (2024)
- [Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART](https://mlsanity.com/paper/2406.07692/) (2024)
- [Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs](https://mlsanity.com/paper/2601.12494/) (2026)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [FASSILA: A Corpus for Algerian Dialect Fake News Detection and Sentiment Analysis](https://mlsanity.com/paper/2411.04604/) (2024)
- [A Tale of Two Scripts: Transliteration and Post-Correction for Judeo-Arabic](https://mlsanity.com/paper/2507.04746/) (2025)
- [AraFinNews: Arabic Financial Summarisation with Domain-Adapted LLMs](https://mlsanity.com/paper/2511.01265/) (2025)
- [AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3](https://mlsanity.com/paper/2512.18399/) (2025)

## Abstract (arXiv)

Like most natural language understanding and generation tasks, state-of-the-art models for summarization are transformer-based sequence-to-sequence architectures that are pretrained on large corpora. While most existing models focused on English, Arabic remained understudied. In this paper we propose AraBART, the first Arabic model in which the encoder and the decoder are pretrained end-to-end, based on BART. We show that AraBART achieves the best performance on multiple abstractive summarization datasets, outperforming strong baselines including a pretrained Arabic BERT-based model and multilingual mBART and mT5 models.