mlsanityData to 14 Oct 2026

LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models

arXiv 2503.16843 ↗First posted 21 Mar 2025cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) excel at generalizing across modalities and tasks, effectively adapting them to specific downstream tasks while simultaneously retaining both general and specialized knowledge remains challenging. Although Low-Rank Adaptation (LoRA) is widely used to efficiently acquire specialized knowledge in MLLMs, it introduces substantial harmful redundancy during visual instruction tuning, which exacerbates the forgetting of general knowledge and degrades downstream task performance. To address this issue, we propose LoRASculpt to eliminate harmful redundant parameters, thereby harmonizing general and specialized knowledge. Specifically, under theoretical guarantees, we introduce sparse updates into LoRA to discard redundant parameters effectively. Furthermore, we propose a Conflict Mitigation Regularizer to refine the update trajectory of LoRA, mitigating knowledge conflicts with the pretrained weights. Extensive experimental results demonstrate that even at very high degree of sparsity (≤ 5%), our method simultaneously enhances generalization and downstream task performance. This confirms that our approach effectively mitigates the catastrophic forgetting issue and further promotes knowledge harmonization in MLLMs.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 84 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

68 of 84 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model6 Mar
  2. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning17 Nov
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models7 Jun
  6. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRA24 Feb
  7. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models19 Feb
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  9. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  5. 2021Learning to Prompt for Continual Learningcited by 944
  6. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  7. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  10. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  13. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  14. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  15. 2024LoRA Learns Less and Forgets Lesscited by 328
  16. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  17. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  18. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  19. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  20. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  21. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  22. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  23. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  24. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  25. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  26. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 67
  27. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  28. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  29. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  30. 2020SpaceNet: Make Free Space For Continual Learningcited by 53
  31. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  32. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  33. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 40
  34. 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
  35. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  36. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  37. 2024Unleashing the Power of Continual Learning on Non-Centralized Devices: A Surveycited by 23
  38. 2022FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregationcited by 21

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
  2. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  3. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 12
  4. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimizationcited by 12
  5. 2026When Attention Betrays: Erasing Backdoor Attacks in Robotic Policies by Reconstructing Visual Tokenscited by 4
  6. 2025ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptationcited by 3
  7. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  8. 2026StAR: Segment Anything Reasonercited by 2
  9. 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptationcited by 1
  10. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  11. 2026RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
  2. 2025From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
  3. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
  4. 2025AsymLoRA: Harmonizing Data Conflicts and Commonalities in MLLMs
  5. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  6. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  7. 2026One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
  8. 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios