mlsanityData to 14 Oct 2026

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

arXiv 2406.05130 ↗First posted 7 Jun 2024cs.CLcited by 22

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) fine-tuned with multimodal instruction datasets have demonstrated remarkable capabilities in multimodal tasks. However, fine-tuning all parameters of MLLMs has become challenging as they usually contain billions of parameters. To address this issue, we study parameter-efficient fine-tuning (PEFT) methods for MLLMs. We aim to identify effective methods for enhancing the performance of MLLMs in scenarios where only a limited number of parameters are trained. This paper conducts empirical studies using four popular PEFT methods to fine-tune the LLM component of open-source MLLMs. We present a comprehensive analysis that encompasses various aspects, including the impact of PEFT methods on various models, parameters and location of the PEFT module, size of fine-tuning data, model stability based on PEFT methods, MLLM's generalization, and hallucination. We evaluated four PEFT methods on seven datasets from two different categories: unseen and seen datasets. Across all experiments, we show that the adapter is the best-performing PEFT method. At the same time, fine-tuning the connector layers leads to improved performance in most MLLMs. Code and data are available at https://github.com/alenai97/PEFT-MLLM.git.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 45 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

28 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
10 more
  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  2. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  3. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  4. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  5. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  6. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  7. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  8. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  9. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  10. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 7.

22 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  2. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  4. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  5. 2024AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robotscited by 7
  6. 2024Demystifying Issues, Causes and Solutions in LLM Open-Source Projectscited by 3
  7. 2025Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Modelscited by 3
  8. 2025Multiple Choice Learning of Low-Rank Adapters for Language Modelingcited by 2
  9. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  10. 2025A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environmentscited by 1
  11. 2025Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learningcited by 1
  12. 2025Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPscited by 1
10 more
  1. 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understandingcited by 1
  2. 2025Research on Driving Scenario Technology Based on Multimodal Large Lauguage Model Optimizationcited by 0
  3. 2025Token-Level Inference-Time Alignment for Vision-Language Modelscited by 0
  4. 2025Optimizing Multimodal Language Models through Attention-based Interpretabilitycited by 0
  5. 2025Selective LoRA for Visual Tokens and Attention Headscited by 0
  6. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruningcited by 0
  7. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  8. 2026Dense Coordinate-List Fine-Tuning Induces a Controllable Interference Surface in Vision-Language Modelscited by 0
  9. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  10. 2026From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  2. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  3. 2024Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models
  4. 2025A Bayesian Hybrid Parameter-Efficient Fine-Tuning Method for Large Language Models
  5. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
  6. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
  7. 2024HFT: Half Fine-Tuning for Large Language Models
  8. 2024Model Composition for Multimodal Large Language Models