# Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond

arXiv: https://arxiv.org/abs/2410.05608 · HTML: https://mlsanity.com/paper/2410.05608/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 5 of 5)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Vision-and-Language Pretrained Models: A Survey](https://mlsanity.com/paper/2204.07356/) (2022)

## Built on by (most cited first, 7 of 7)

- [TerraMind: Large-Scale Generative Multimodality for Earth Observation](https://mlsanity.com/paper/2504.11171/) (2025)
- [COP-GEN-Beta: Unified Generative Modelling of COPernicus Imagery Thumbnails](https://mlsanity.com/paper/2504.08548/) (2025)
- [Evaluating Graphical Perception with Multimodal LLMs](https://mlsanity.com/paper/2504.04221/) (2025)
- [From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement](https://mlsanity.com/paper/2512.21598/) (2025)
- [Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuning](https://mlsanity.com/paper/2601.16419/) (2026)
- [Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach](https://mlsanity.com/paper/2503.04918/) (2025)
- [Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models](https://mlsanity.com/paper/2606.13624/) (2026)

## Abstract (arXiv)

This tutorial explores recent advancements in multimodal pretrained and large models, capable of integrating and processing diverse data forms such as text, images, audio, and video. Participants will gain an understanding of the foundational concepts of multimodality, the evolution of multimodal research, and the key technical challenges addressed by these models. We will cover the latest multimodal datasets and pretrained models, including those beyond vision and language. Additionally, the tutorial will delve into the intricacies of multimodal large models and instruction tuning strategies to optimise performance for specific tasks. Hands-on laboratories will offer practical experience with state-of-the-art multimodal models, demonstrating real-world applications like visual storytelling and visual question answering. This tutorial aims to equip researchers, practitioners, and newcomers with the knowledge and skills to leverage multimodal AI. ACM Multimedia 2024 is the ideal venue for this tutorial, aligning perfectly with our goal of understanding multimodal pretrained and large language models, and their tuning mechanisms.