mlsanityData to 14 Oct 2026

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

arXiv 2504.07448 ↗First posted 10 Apr 2025cs.LG cs.AI cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices A as random projections and sparsifies the matrices B using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https://github.com/juzhengz/LoRI

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 69 of its 81 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

69 of 81 references.

References · 69

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023Mistral 7Bcited by 3,164
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2016Learning without Forgettingcited by 2,496
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2016Progressive Neural Networkscited by 1,361
  11. 2017Continual Learning with Deep Generative Replaycited by 1,158
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  13. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  14. 2022Editing Models with Task Arithmeticcited by 1,120
  15. 2018Experience Replay for Continual Learningcited by 1,095
  16. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  17. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  18. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  19. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  20. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  21. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  22. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  23. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  24. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  25. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  26. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  27. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  28. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  29. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  30. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  31. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  32. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  33. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  34. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  35. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  36. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  37. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  38. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  2. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 18
  3. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  4. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 5
  5. 2025TARA: Token-Aware LoRA for Composable Personalization in Diffusion Modelscited by 5
  6. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  7. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  8. 2025Towards Efficient Multimodal Unified Reasoning Model via Model Mergingcited by 3
  9. 2025DiaBlo: Diagonal Blocks Are Sufficient For Finetuningcited by 2
  10. 2026Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRAcited by 2
  11. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  12. 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?cited by 1
7 more
  1. 2026REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modelingcited by 1
  2. 2025Deep Generative Continual Learning using Functional LoRA: FunLoRAcited by 0
  3. 2026Modular Multi-Task Learning for Chemical Reaction Predictioncited by 0
  4. 2026Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probescited by 0
  5. 2026AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environmentscited by 0
  6. 2026Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuningcited by 0
  7. 2026Socio-Foundation: A Model for Generalizable Individual Behavior Simulation via Hierarchical Capability Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  2. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  3. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
  4. 2026TLoRA: Task-aware Low Rank Adaptation of Large Language Models
  5. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  6. 2025From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
  7. 2026ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
  8. 2026Parameter-Efficient Fine-Tuning with Learnable Rank