mlsanityData to 7 Oct 2026

Selective LoRA for Visual Tokens and Attention Heads

arXiv 2512.19219 ↗First posted 22 Dec 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Low-rank adaptation (LoRA) is widely used for parameter-efficient fine-tuning, but its standard all-token, all-head design ignores the heterogeneous structure of vision language model (VLM) inputs. We introduce Image-LoRA, a vision-oriented PEFT recipe that views LoRA as a token-level residual update and applies this update only to visual tokens. Image-LoRA further restricts adaptation to the value path of a compact subset of attention heads, selected using a one-pass influence estimate from a rank-1 visual-token-only probe. This token-, head-, and value-selective design reduces trainable parameters and adapter-only training FLOPs while leaving the pure-text forward pass of the frozen backbone unchanged when no visual tokens are present. Across visual localization benchmarks with controlled text:image token ratios, Image-LoRA matches or closely approaches standard LoRA, while showing especially favorable trade-offs in image-token-heavy regimes. We further validate its generality on TextVQA and VideoQA, verify pure-text preservation on GSM8K, and show on ViLP that a stronger information bottleneck can yield gains over standard LoRA.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 44 references that are in the map. First posted on arXiv 22 Dec 2025.

Builds on

39 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models28 May

Other references · 38

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  9. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2019Importance Estimation for Neural Network Pruningcited by 531
  12. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
26 more
  1. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  3. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  4. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  5. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  6. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  7. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  8. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
  9. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  10. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  11. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  12. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  13. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  14. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  15. 2025Glyph: Scaling Context Windows via Visual-Text Compressioncited by 49
  16. 2024Probing Visual Language Priors in VLMscited by 44
  17. 2025Vision as LoRAcited by 37
  18. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
  19. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  20. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  21. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  22. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Headscited by 9
  23. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  24. 2024Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuningcited by 6
  25. 2025MokA: Multimodal Low-Rank Adaptation for MLLMscited by 6
  26. 2025GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devicescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models
  2. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models
  3. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
  4. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation
  5. 2024Empower Vision Applications with LoRA LMM
  6. 2026MuRA: Multi-Rank Adaptation for Efficient and Effective Test-Time Vision-Language Generalization
  7. 2025LoRAtorio: An intrinsic approach to LoRA Skill Composition
  8. 2025Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification