mlsanityData to 7 Oct 2026

Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages

arXiv 2605.17152 ↗First posted 16 May 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal LLMs are evolving from vision-language to tri-modality that see, hear, and read, yet pipelines and benchmarks remain English-centric and compute-heavy. The tutorial offers an overview of this emerging research area for multilingual multimodality across text, speech, and vision under limited data/compute budgets, synthesizing foundations, recent multilingual models (PALO, Maya), speech-text LLMs. We cover low-cost data creation/curation; adapter stacks for tri-modal alignment; culture-aware evaluation beyond English and hands on resources for fine-tuning a compact multilingual VLM and wiring a speech->text->LLM pipeline. The content will be delivered as an interactive half-day tutorial, designed for researchers and practitioners working on multilingual, multimodal AI in low-resource language settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 30 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

15 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Large Multimodal Models for Low-Resource Languages: A Survey8 Feb
  2. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  8. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  9. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  10. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 105
  11. 2024Maya: An Instruction Finetuned Multilingual Multimodal Modelcited by 15
  12. 2023HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Languagecited by 12
1 more
  1. 2025DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboardscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  2. 2024LLMs Meet Multimodal Generation and Editing: A Survey
  3. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
  7. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  8. 2024Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages