mlsanityData to 14 Oct 2026

An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

arXiv 2309.09958 ↗First posted 18 Sep 2023cs.CV cs.CLcited by 32

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. Additionally, the study highlights the importance of higher image resolutions and mixing multimodal-language data to improve LMM performance, and visual instruction tuning can sometimes improve LMM's pure language capability. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible, thus helping establish stronger baselines for future research. Code and checkpoints will be made public.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 22 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

17 of 22 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial26 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 12

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 18; 2025: 5; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  4. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  5. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  6. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  7. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 63
  8. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 58
  9. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  10. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 46
  11. 2024FineVQ: Fine-Grained User Generated Content Video Quality Assessmentcited by 28
  12. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Modelscited by 24
20 more
  1. 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Modelscited by 20
  2. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  3. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  4. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanesecited by 12
  5. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Modelscited by 11
  6. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  7. 2026Recursive Multi-Agent Systemscited by 10
  8. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reportscited by 6
  9. 2025Transport Discrepancy as a Reliability Signal for Vision-Language-Action Modelscited by 6
  10. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusioncited by 5
  11. 2025Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrievalcited by 5
  12. 2024Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunitiescited by 3
  13. 2025SAM2LoRA: Composite Loss-Guided, Parameter-Efficient Finetuning of SAM2 for Retinal Fundus Segmentationcited by 3
  14. 2024Foundations of Multisensory Artificial Intelligencecited by 1
  15. 2024UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languagescited by 1
  16. 2026Balancing Image Compression and Generation with Bootstrapped Tokenizationcited by 1
  17. 2024Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Networkcited by 0
  18. 2024Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Promptscited by 0
  19. 2025PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapterscited by 0
  20. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  2. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  3. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
  4. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  5. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
  6. 2023On the Performance of Multimodal Language Models
  7. 2024Error-driven Data-efficient Large Multimodal Model Tuning
  8. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models