mlsanityData to 7 Oct 2026

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

arXiv 2310.12100 ↗First posted 18 Oct 2023cs.CL cs.AI cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large language models (LLMs) and vision language models (VLMs) demonstrate excellent performance on a wide range of tasks by scaling up parameter counts from O(10^9) to O(10^{12}) levels and further beyond. These large scales make it impossible to adapt and deploy fully specialized models given a task of interest. Parameter-efficient fine-tuning (PEFT) emerges as a promising direction to tackle the adaptation and serving challenges for such large models. We categorize PEFT techniques into two types: intrusive and non-intrusive. Intrusive PEFT techniques directly change a model's internal architecture. Though more flexible, they introduce significant complexities for training and serving. Non-intrusive PEFT techniques leave the internal architecture unchanged and only adapt model-external parameters, such as embeddings for input. In this work, we describe AdaLink as a non-intrusive PEFT technique that achieves competitive performance compared to SoTA intrusive PEFT (LoRA) and full model fine-tuning (FT) on various tasks. We evaluate using both text-only and multimodal tasks, with experiments that account for both parameter-count scaling and training regime (with and without instruction tuning).

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 46 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

46 of 46 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
34 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  16. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  18. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  20. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  21. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  22. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  23. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  24. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  25. 2019Scene Text Visual Question Answeringcited by 327
  26. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  27. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  28. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  29. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  30. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  31. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  32. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  33. 2022Prompt Tuning for Generative Multimodal Pretrained Modelscited by 25
  34. 2021AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapterscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 6; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  3. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  4. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  5. 2024Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Usecited by 9
  6. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reportscited by 6
  7. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  8. 2025PaCA: Partial Connection Adaptation for Efficient Fine-Tuningcited by 6
  9. 2024Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningcited by 4
  10. 2024Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuningcited by 4
  11. 2025LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  12. 2025LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysiscited by 4
7 more
  1. 2025LoRA Is Slower Than You Thinkcited by 3
  2. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 3
  3. 2024SBoRA: Low-Rank Adaptation with Regional Weight Updatescited by 2
  4. 2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2
  5. 2024Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Networkcited by 0
  6. 2025MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMscited by 0
  7. 2026PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Promptscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  2. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models
  3. 2023Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation
  4. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  5. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
  6. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  7. 2024Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models
  8. 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning