mlsanityData to 7 Oct 2026

PALO: A Polyglot Large Multimodal Model for 5B People

arXiv 2402.14818 ↗First posted 22 Feb 2024cs.CL cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In pursuit of more inclusive Vision-Language Models (VLMs), this study introduces a Large Multilingual Multimodal Model called PALO. PALO offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 30 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

22 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning12 Oct
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
7 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  5. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  6. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  7. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 20; 2026: 7.

34 papers.

Sort:

By area:

  1. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  2. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  3. 2024Dallah: A Dialect-Aware Multimodal Large Language Model for Arabiccited by 16
  4. 2024Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russiancited by 12
  5. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  6. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Modelscited by 10
  7. 2025Chitrarth: Bridging Vision and Language for a Billion Peoplecited by 10
  8. 2024mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpuscited by 8
  9. 2025JEEM: Vision-Language Understanding in Four Arabic Dialectscited by 7
  10. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7
  11. 2025Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translationcited by 6
  12. 2025MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understandingcited by 6
22 more
  1. 2025GAEA: A Geolocation Aware Conversational Assistantcited by 5
  2. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  3. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  4. 2024Mitigating Multilingual Hallucination in Large Vision-Language Modelscited by 3
  5. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  6. 2025Behind Maya: Building a Multilingual Vision Language Modelcited by 3
  7. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  8. 2025Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuningcited by 3
  9. 2026When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Modelscited by 3
  10. 2025Synthetic Document Question Answering in Hungariancited by 1
  11. 2025Multilingual Vision-Language Models, A Surveycited by 1
  12. 2025Seeing Straight: Document Orientation Detection for Efficient OCRcited by 1
  13. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  14. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  15. 2025NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledgecited by 0
  16. 2025HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samplescited by 0
  17. 2025Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Modelscited by 0
  18. 2026Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languagescited by 0
  19. 2026"Inctelegi Romanecste?'' A Recipe for Romanian Vision-Language Modelscited by 0
  20. 2026Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Modelscited by 0
  21. 2026Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluationcited by 0
  22. 2026One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR
  2. 2026Multilingual Training and Evaluation Resources for Vision-Language Models
  3. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  4. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  7. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models