# PALO: A Polyglot Large Multimodal Model for 5B People

arXiv: https://arxiv.org/abs/2402.14818 · HTML: https://mlsanity.com/paper/2402.14818/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning](https://mlsanity.com/paper/2310.08166/) (2023)

## Built on by (most cited first, 30 of 34)

- [Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages](https://mlsanity.com/paper/2410.16153/) (2024)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic](https://mlsanity.com/paper/2407.18129/) (2024)
- [Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian](https://mlsanity.com/paper/2405.13929/) (2024)
- [Why do LLaVA Vision-Language Models Reply to Images in English?](https://mlsanity.com/paper/2407.02333/) (2024)
- [BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models](https://mlsanity.com/paper/2502.08779/) (2025)
- [Chitrarth: Bridging Vision and Language for a Billion People](https://mlsanity.com/paper/2502.15392/) (2025)
- [mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus](https://mlsanity.com/paper/2406.08707/) (2024)
- [IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs](https://mlsanity.com/paper/2511.04727/) (2025)
- [JEEM: Vision-Language Understanding in Four Arabic Dialects](https://mlsanity.com/paper/2503.21910/) (2025)
- [MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understanding](https://mlsanity.com/paper/2505.20298/) (2025)
- [Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation](https://mlsanity.com/paper/2502.20420/) (2025)
- [GAEA: A Geolocation Aware Conversational Assistant](https://mlsanity.com/paper/2503.16423/) (2025)
- [OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA](https://mlsanity.com/paper/2510.06371/) (2025)
- [Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization](https://mlsanity.com/paper/2503.22577/) (2025)
- [When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models](https://mlsanity.com/paper/2606.11906/) (2026)
- [Mitigating Multilingual Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2408.00550/) (2024)
- [Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model](https://mlsanity.com/paper/2501.05122/) (2025)
- [Behind Maya: Building a Multilingual Vision Language Model](https://mlsanity.com/paper/2505.08910/) (2025)
- [POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering](https://mlsanity.com/paper/2507.11939/) (2025)
- [Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning](https://mlsanity.com/paper/2509.16628/) (2025)
- [Synthetic Document Question Answering in Hungarian](https://mlsanity.com/paper/2505.23008/) (2025)
- [Multilingual Vision-Language Models, A Survey](https://mlsanity.com/paper/2509.22123/) (2025)
- [Seeing Straight: Document Orientation Detection for Efficient OCR](https://mlsanity.com/paper/2511.04161/) (2025)
- [M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG](https://mlsanity.com/paper/2512.05959/) (2025)
- [DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging](https://mlsanity.com/paper/2605.12960/) (2026)
- [NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledge](https://mlsanity.com/paper/2504.05995/) (2025)
- [HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples](https://mlsanity.com/paper/2511.15183/) (2025)
- [Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models](https://mlsanity.com/paper/2512.14926/) (2025)
- [Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages](https://mlsanity.com/paper/2603.23521/) (2026)

## Abstract (arXiv)

In pursuit of more inclusive Vision-Language Models (VLMs), this study introduces a Large Multilingual Multimodal Model called PALO. PALO offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.