# Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning

arXiv: https://arxiv.org/abs/2302.11289 · HTML: https://mlsanity.com/paper/2302.11289/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Learning to Branch for Multi-Task Learning](https://mlsanity.com/paper/2006.01895/) (2020)
- [RotoGrad: Gradient Homogenization in Multitask Learning](https://mlsanity.com/paper/2103.02631/) (2021)
- [A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks](https://mlsanity.com/paper/1611.01587/) (2016)
- [Indoor Semantic Segmentation using depth information](https://mlsanity.com/paper/1301.3572/) (2013)
- [Automated Search for Resource-Efficient Branched Multi-Task Networks](https://mlsanity.com/paper/2008.10292/) (2020)
- [MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning](https://mlsanity.com/paper/2003.14058/) (2020)

## Built on by (most cited first, 30 of 51)

- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent](https://mlsanity.com/paper/2501.01230/) (2025)
- [Understanding Layer Significance in LLM Alignment](https://mlsanity.com/paper/2410.17875/) (2024)
- [Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models](https://mlsanity.com/paper/2509.24365/) (2025)
- [HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.18080/) (2024)
- [Gradient Surgery for One-shot Unlearning on Generative Model](https://mlsanity.com/paper/2307.04550/) (2023)
- [GRIDS: Grouped Multiple-Degradation Restoration with Image Degradation Similarity](https://mlsanity.com/paper/2407.12273/) (2024)
- [Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective](https://mlsanity.com/paper/2411.18615/) (2024)
- [Quantifying Task Priority for Multi-Task Optimization](https://mlsanity.com/paper/2406.02996/) (2024)
- [Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2410.02681/) (2024)
- [FREE: Faster and Better Data-Free Meta-Learning](https://mlsanity.com/paper/2405.00984/) (2024)
- [SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training](https://mlsanity.com/paper/2601.23155/) (2026)
- [Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning](https://mlsanity.com/paper/2408.07985/) (2024)
- [Gradient Similarity Surgery in Multi-Task Deep Learning](https://mlsanity.com/paper/2506.06130/) (2025)
- [Scalable Multi-Objective Robot Reinforcement Learning through Gradient Conflict Resolution](https://mlsanity.com/paper/2509.14816/) (2025)
- [Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learning](https://mlsanity.com/paper/2507.07485/) (2025)
- [SLIM: Skill Learning with Multiple Critics](https://mlsanity.com/paper/2402.00823/) (2024)
- [Towards Principled Task Grouping for Multi-Task Learning](https://mlsanity.com/paper/2402.15328/) (2024)
- [Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradients](https://mlsanity.com/paper/2410.02845/) (2024)
- [A Multi-Task Learning Approach to Linear Multivariate Forecasting](https://mlsanity.com/paper/2502.03571/) (2025)
- [One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning](https://mlsanity.com/paper/2509.07945/) (2025)
- [FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning](https://mlsanity.com/paper/2310.13746/) (2023)
- [Examining Common Paradigms in Multi-Task Learning](https://mlsanity.com/paper/2311.04698/) (2023)
- [On The Fairness Impacts of Hardware Selection in Machine Learning](https://mlsanity.com/paper/2312.03886/) (2023)
- [Nested Hash Layer: A Plug-and-play Module for Multiple-length Hash Code Learning](https://mlsanity.com/paper/2412.08922/) (2024)
- [Domain Generalization via Pareto Optimal Gradient Matching](https://mlsanity.com/paper/2507.14227/) (2025)
- [Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning](https://mlsanity.com/paper/2507.21049/) (2025)
- [GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning](https://mlsanity.com/paper/2509.07252/) (2025)
- [Uncovering Cross-Objective Interference in Multi-Objective Alignment](https://mlsanity.com/paper/2602.06869/) (2026)
- [AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent](https://mlsanity.com/paper/2404.07428/) (2024)

## Abstract (arXiv)

A fundamental challenge for multi-task learning is that different tasks may conflict with each other when they are solved jointly, and a cause of this phenomenon is conflicting gradients during optimization. Recent works attempt to mitigate the influence of conflicting gradients by directly altering the gradients based on some criteria. However, our empirical study shows that ``gradient surgery'' cannot effectively reduce the occurrence of conflicting gradients. In this paper, we take a different approach to reduce conflicting gradients from the root. In essence, we investigate the task gradients w.r.t. each shared network layer, select the layers with high conflict scores, and turn them to task-specific layers. Our experiments show that such a simple approach can greatly reduce the occurrence of conflicting gradients in the remaining shared layers and achieve better performance, with only a slight increase in model parameters in many cases. Our approach can be easily applied to improve various state-of-the-art methods including gradient manipulation methods and branched architecture search methods. Given a network architecture (e.g., ResNet18), it only needs to search for the conflict layers once, and the network can be modified to be used with different methods on the same or even different datasets to gain performance improvement. The source code is available at https://github.com/moukamisama/Recon.