# Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation

arXiv: https://arxiv.org/abs/2310.16127 · HTML: https://mlsanity.com/paper/2310.16127/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Neural Arabic Question Answering](https://mlsanity.com/paper/1906.05394/) (2019)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)
- [AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://mlsanity.com/paper/2203.10945/) (2022)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [Arabic Text Diacritization Using Deep Neural Networks](https://mlsanity.com/paper/1905.01965/) (2019)
- [Machine Generation and Detection of Arabic Manipulated and Fake News](https://mlsanity.com/paper/2011.03092/) (2020)
- [AraNet: A Deep Learning Toolkit for Arabic Social Media](https://mlsanity.com/paper/1912.13072/) (2019)
- [TURJUMAN: A Public Toolkit for Neural Arabic Machine Translation](https://mlsanity.com/paper/2206.03933/) (2022)
- [Design Challenges in Named Entity Transliteration](https://mlsanity.com/paper/1808.02563/) (2018)
- [ANETAC: Arabic Named Entity Transliteration and Classification Dataset](https://mlsanity.com/paper/1907.03110/) (2019)

## Built on by (most cited first, 6 of 6)

- [M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Models](https://mlsanity.com/paper/2406.16783/) (2024)
- [Exploiting Dialect Identification in Automatic Dialectal Text Normalization](https://mlsanity.com/paper/2407.03020/) (2024)
- [Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis](https://mlsanity.com/paper/2405.14277/) (2024)
- [Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer](https://mlsanity.com/paper/2505.10945/) (2025)
- [AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation](https://mlsanity.com/paper/2609.22796/) (2026)
- [SinaTools: Open Source Toolkit for Arabic Natural Language Processing](https://mlsanity.com/paper/2411.01523/) (2024)

## Abstract (arXiv)

Understanding Arabic text and generating human-like responses is a challenging endeavor. While many researchers have proposed models and solutions for individual problems, there is an acute shortage of a comprehensive Arabic natural language generation toolkit that is capable of handling a wide range of tasks. In this work, we present a novel Arabic text-to-text Transformer model, namely AraT5v2. Our new model is methodically trained on extensive and diverse data, utilizing an extended sequence length of 2,048 tokens. We explore various pretraining strategies including unsupervised, supervised, and joint pertaining, under both single and multitask settings. Our models outperform competitive baselines with large margins. We take our work one step further by developing and publicly releasing Octopus, a Python-based package and command-line toolkit tailored for eight Arabic generation tasks all exploiting a single model. We release the models and the toolkit on our public repository.