mlsanityData to 14 Oct 2026

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models

arXiv 2403.15226 ↗First posted 22 Mar 2024cs.MM cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we propose a novel parameter and computation efficient tuning method for Multi-modal Large Language Models (MLLMs), termed Efficient Attention Skipping (EAS). Concretely, we first reveal that multi-head attentions (MHAs), the main computational overhead of MLLMs, are often redundant to downstream tasks. Based on this observation, EAS evaluates the attention redundancy and skips the less important MHAs to speed up inference. Besides, we also propose a novel propagation-of-information adapter (PIA) to serve the attention skipping of EAS and keep parameter efficiency, which can be further re-parameterized into feed-forward networks (FFNs) for zero-extra latency. To validate EAS, we apply it to a recently proposed MLLM called LaVIN and a classic VL pre-trained model called METER, and conduct extensive experiments on a set of benchmarks. The experiments show that EAS not only retains high performance and parameter efficiency, but also greatly speeds up inference speed. For instance, LaVIN-EAS can obtain 89.98% accuracy on ScineceQA while speeding up inference by 2.2 times to LaVIN

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 85 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

73 of 85 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2022Visual Prompt Tuningcited by 1,681
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  13. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  14. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  15. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  16. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  20. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  21. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  24. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  25. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  26. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  27. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  28. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  29. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  30. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  31. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  32. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  33. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  34. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  35. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  36. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  37. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  38. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129

50 most cited of 69 All 69 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Skipping Computations in Multimodal LLMs
  2. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  3. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
  4. 2024TAIA: Large Language Models are Out-of-Distribution Data Learners
  5. 2026Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion
  6. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
  7. 2025Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
  8. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models