mlsanityData to 7 Oct 2026

Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

arXiv 2410.16153 ↗First posted 21 Oct 2024cs.CL cs.CVcited by 63

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Despite recent advances in multimodal large language models (MLLMs), their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world's languages and diverse cultural contexts underrepresented. This paper introduces Pangea, a multilingual multimodal LLM trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages. PangeaIns features: 1) high-quality English instructions, 2) carefully machine-translated instructions, and 3) culturally relevant multimodal tasks to ensure cross-cultural coverage. To rigorously assess models' capabilities, we introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets covering 47 languages. Results show that Pangea significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Ablation studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall performance. We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across a broader linguistic and cultural spectrum.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 82 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

43 of 82 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese22 Aug
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  4. 2024PALO: A Polyglot Large Multimodal Model for 5B People22 Feb
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
25 more
  1. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2021OCR-free Document Understanding Transformercited by 400
  5. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  6. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  7. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  8. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  9. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  11. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  12. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  13. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  14. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  15. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  16. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  17. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  18. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  19. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  20. 2024An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevancecited by 36
  21. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  22. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  23. 2022MaXM: Towards Multilingual Visual Question Answeringcited by 9
  24. 2023Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Normscited by 8
  25. 2022ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 40; 2026: 19.

63 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  3. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  4. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluationcited by 19
  5. 2024CAMEL-Bench: A Comprehensive Arabic LMM Benchmarkcited by 17
  6. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledgecited by 16
  7. 2025TigerLLM - A Family of Bangla Large Language Modelscited by 15
  8. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  9. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  10. 2025AIN: The Arabic INclusive Large Multimodal Modelcited by 8
  11. 2025Multimodal Cultural Safety: Evaluation Framework and Alignment Strategiescited by 8
  12. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
38 more
  1. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  2. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7
  3. 2024VARCO-VISION: Expanding Frontiers in Korean Vision-Language Modelscited by 6
  4. 2025MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understandingcited by 6
  5. 2025KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contextscited by 6
  6. 2025TigerCoder: A Novel Suite of LLMs for Code Generation in Banglacited by 6
  7. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matchingcited by 5
  8. 2025VARCO-VISION-2.0 Technical Reportcited by 5
  9. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  10. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  11. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  12. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  13. 2025xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillationcited by 3
  14. 2025Behind Maya: Building a Multilingual Vision Language Modelcited by 3
  15. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  16. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  17. 2025Finding Culture-Sensitive Neurons in Vision-Language Modelscited by 3
  18. 2025Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Timescited by 2
  19. 2025NeoBabel: A Multilingual Open Tower for Visual Generationcited by 2
  20. 2026Solar Open Technical Reportcited by 2
  21. 2026M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Modelscited by 2
  22. 2026Anthropogenic Regional Adaptation in Multimodal Vision-Language Modelcited by 2
  23. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modelingcited by 1
  24. 2025Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVAcited by 1
  25. 2025CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluationcited by 1
  26. 2025Multilingual Vision-Language Models, A Surveycited by 1
  27. 2025TowerVision: Understanding and Improving Multilinguality in Vision-Language Modelscited by 1
  28. 2025Seeing Straight: Document Orientation Detection for Efficient OCRcited by 1
  29. 2025Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tablescited by 1
  30. 2025jina-vlm: Small Multilingual Vision Language Modelcited by 1
  31. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  32. 2026MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoningcited by 1
  33. 2026Beyond Factual QA: Mentorship-Oriented Question Answering over Long-Form Multilingual Contentcited by 1
  34. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  35. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  36. 2026Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learningcited by 1
  37. 2025NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledgecited by 0
  38. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioningcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  2. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  3. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  4. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  5. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
  6. 2024Multilingual Large Language Models: A Systematic Survey
  7. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
  8. 2024Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages