# Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection

arXiv: https://arxiv.org/abs/2601.09684 · HTML: https://mlsanity.com/paper/2601.09684/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 13 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts](https://mlsanity.com/paper/2510.08396/) (2025)
- [Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptation](https://mlsanity.com/paper/2601.07935/) (2026)

## Built on by (most cited first, 8 of 8)

- [Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation](https://mlsanity.com/paper/2603.02224/) (2026)
- [Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs](https://mlsanity.com/paper/2609.02548/) (2026)
- [Representation Collapse in Sequential Post-Training of Large Language Models](https://mlsanity.com/paper/2605.30524/) (2026)
- [Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback](https://mlsanity.com/paper/2606.14368/) (2026)
- [Online Data Selection Is Implicit Alignment](https://mlsanity.com/paper/2607.07023/) (2026)
- [Automated Gradient-Driven Parameter Sharing for Low-Resource Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2603.25836/) (2026)
- [Parameter-efficient Quantum Multi-task Learning](https://mlsanity.com/paper/2604.13560/) (2026)
- [Interference-Aware Multi-Task Unlearning](https://mlsanity.com/paper/2605.19042/) (2026)

## Abstract (arXiv)

Multi-Task Learning (MTL) combined with Low-Rank Adaptation (LoRA) has emerged as a promising direction for parameter-efficient deployment of Large Language Models (LLMs). By sharing a single adapter across multiple tasks, one can significantly reduce storage overhead. However, this approach suffers from negative transfer, where conflicting gradient updates from distinct tasks degrade the performance of individual tasks compared to single-task fine-tuning. This problem is exacerbated in LoRA due to the low-rank constraint, which limits the optimization landscape's capacity to accommodate diverse task requirements. In this paper, we propose Ortho-LoRA, a gradient projection method specifically tailored for the bipartite structure of LoRA. Ortho-LoRA dynamically projects conflicting task gradients onto the orthogonal complement of each other within the intrinsic LoRA subspace. Extensive experiments on the GLUE benchmark demonstrate that Ortho-LoRA effectively mitigates task interference, outperforming standard joint training and recovering 95% of the performance gap between multi-task and single-task baselines with negligible computational overhead.