mlsanityData to 7 Oct 2026

An Empirical Study of Parameter Efficient Fine-tuning on Vision-Language Pre-train Model

arXiv 2403.08433 ↗First posted 13 Mar 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent studies applied Parameter Efficient Fine-Tuning techniques (PEFTs) to efficiently narrow the performance gap between pre-training and downstream. There are two important factors for various PEFTs, namely, the accessible data size and fine-tunable parameter size. A natural expectation for PEFTs is that the performance of various PEFTs is positively related to the data size and fine-tunable parameter size. However, according to the evaluation of five PEFTs on two downstream vision-language (VL) tasks, we find that such an intuition holds only if the downstream data and task are not consistent with pre-training. For downstream fine-tuning consistent with pre-training, data size no longer affects the performance, while the influence of fine-tunable parameter size is not monotonous. We believe such an observation could guide the choice of training strategy for various PEFTs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 30 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

24 of 30 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
12 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  8. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  9. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  10. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42
  11. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  12. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  2. 2024Rethinking Overlooked Aspects in Vision-Language Models
  3. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
  4. 2024Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
  5. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  6. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  7. 2023Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation
  8. 2024Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification