# Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks

arXiv: https://arxiv.org/abs/2403.01031 · HTML: https://mlsanity.com/paper/2403.01031/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training](https://mlsanity.com/paper/2110.14883/) (2021)
- [mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs](https://mlsanity.com/paper/2307.06930/) (2023)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)
- [Machine Generation and Detection of Arabic Manipulated and Fake News](https://mlsanity.com/paper/2011.03092/) (2020)
- [TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://mlsanity.com/paper/2308.03051/) (2023)

## Built on by (most cited first, 30 of 32)

- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://mlsanity.com/paper/2509.02550/) (2025)
- [CAMEL-Bench: A Comprehensive Arabic LMM Benchmark](https://mlsanity.com/paper/2410.18976/) (2024)
- [Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy](https://mlsanity.com/paper/2412.17759/) (2024)
- [Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic](https://mlsanity.com/paper/2407.18129/) (2024)
- [KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding](https://mlsanity.com/paper/2502.14949/) (2025)
- [Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs](https://mlsanity.com/paper/2503.00151/) (2025)
- [The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology](https://mlsanity.com/paper/2506.01340/) (2025)
- [Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifacts](https://mlsanity.com/paper/2502.14865/) (2025)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs](https://mlsanity.com/paper/2511.04727/) (2025)
- [Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition](https://mlsanity.com/paper/2407.13559/) (2024)
- [JEEM: Vision-Language Understanding in Four Arabic Dialects](https://mlsanity.com/paper/2503.21910/) (2025)
- [LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?](https://mlsanity.com/paper/2503.15003/) (2025)
- [Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset](https://mlsanity.com/paper/2505.21979/) (2025)
- [LaVy: Vietnamese Multimodal Large Language Model](https://mlsanity.com/paper/2404.07922/) (2024)
- [QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation](https://mlsanity.com/paper/2506.02295/) (2025)
- [Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale](https://mlsanity.com/paper/2509.14008/) (2025)
- [Large Multimodal Models for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2502.05568/) (2025)
- [ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark](https://mlsanity.com/paper/2505.17021/) (2025)
- [Isolating Culture Neurons in Multilingual Large Language Models](https://mlsanity.com/paper/2508.02241/) (2025)
- [From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher](https://mlsanity.com/paper/2507.10216/) (2025)
- [Arabic Automatic Story Generation with Large Language Models](https://mlsanity.com/paper/2407.07551/) (2024)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://mlsanity.com/paper/2508.12227/) (2025)
- [ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning](https://mlsanity.com/paper/2602.17054/) (2026)
- [VQA support to Arabic Language Learning Educational Tool](https://mlsanity.com/paper/2508.03488/) (2025)
- [Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora](https://mlsanity.com/paper/2511.07080/) (2025)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have proven effective in a wide range of tasks requiring complex reasoning and linguistic comprehension. However, due to a lack of high-quality multimodal resources in languages other than English, success of MLLMs remains relatively limited to English-based settings. This poses significant challenges in developing comparable models for other languages, including even those with large speaker populations such as Arabic. To alleviate this challenge, we introduce a comprehensive family of Arabic MLLMs, dubbed textitPeacock, with strong vision and language capabilities. Through comprehensive qualitative and quantitative analysis, we demonstrate the solid performance of our models on various visual reasoning tasks and further show their emerging dialectal potential. Additionally, we introduce ~textitHenna, a new benchmark specifically designed for assessing MLLMs on aspects related to Arabic culture, setting the first stone for culturally-aware Arabic MLLMs.The GitHub repository for the textitPeacock project is available at urlhttps://github.com/UBC-NLP/peacock.