mlsanityData to 7 Oct 2026

MMMModal -- Multi-Images Multi-Audio Multi-turn Multi-Modal

arXiv 2402.11297 ↗First posted 17 Feb 2024cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Our contribution introduces a groundbreaking multimodal large language model designed to comprehend multi-images, multi-audio, and multi-images-multi-audio within a single multiturn session. Leveraging state-of-the-art models, we utilize the SigLIP encoder for visual inputs and the Whisper Encoder for audio inputs. Notably, this multimodal large language model is bilingual, proficient in understanding both English and Malay simultaneously. We proudly unveil two versions of this model: TinyLlama with 1.1B parameters, and Mistral with 7B parameters. With its ability to navigate diverse modalities and languages, our model represents a significant advancement for the Malaysian context and beyond. All models released at https://huggingface.co/collections/mesolitica/multimodal-malaysian-llm-65c6f893e03f78fa9e5c8859

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 1 of its 6 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

1 of 6 references.

References · 1

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  2. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentationcited by 13

Similar, not cited

Close in wording, no citation either way

  1. 2023Multimodal Large Language Models: A Survey
  2. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  3. 2024A Review of Multi-Modal Large Language and Vision Models
  4. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
  5. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  7. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training