# Gradient Similarity Surgery in Multi-Task Deep Learning

arXiv: https://arxiv.org/abs/2506.06130 · HTML: https://mlsanity.com/paper/2506.06130/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](https://mlsanity.com/paper/2111.10603/) (2021)
- [FAMO: Fast Adaptive Multitask Optimization](https://mlsanity.com/paper/2306.03792/) (2023)
- [Independent Component Alignment for Multi-Task Learning](https://mlsanity.com/paper/2305.19000/) (2023)
- [Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning](https://mlsanity.com/paper/2302.11289/) (2023)
- [Variational Multi-Task Learning with Gumbel-Softmax Priors](https://mlsanity.com/paper/2111.05323/) (2021)
- [GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning](https://mlsanity.com/paper/2301.13465/) (2023)
- [Multi-task Learning for Real-time Autonomous Driving Leveraging Task-adaptive Attention Generator](https://mlsanity.com/paper/2403.03468/) (2024)
- [Retinal Structure Detection in OCTA Image via Voting-based Multi-task Learning](https://mlsanity.com/paper/2208.10745/) (2022)
- [Constructing Time-Series Momentum Portfolios with Deep Multi-Task Learning](https://mlsanity.com/paper/2306.13661/) (2023)
- [Cross-Task Attention Network: Improving Multi-Task Learning for Medical Imaging Applications](https://mlsanity.com/paper/2309.03837/) (2023)
- [Examining Common Paradigms in Multi-Task Learning](https://mlsanity.com/paper/2311.04698/) (2023)

## Built on by (most cited first, 5 of 5)

- [Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models](https://mlsanity.com/paper/2510.21978/) (2025)
- [GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning](https://mlsanity.com/paper/2509.07252/) (2025)
- [Graph Coloring for Multi-Task Learning](https://mlsanity.com/paper/2509.16959/) (2025)
- [XekRung Technical Report](https://mlsanity.com/paper/2605.00072/) (2026)
- [Gradient Surgery for Physics-Informed Neural Networks](https://mlsanity.com/paper/2609.30966/) (2026)

## Abstract (arXiv)

The multi-task learning (MTL) paradigm aims to simultaneously learn multiple tasks within a single model capturing higher-level, more general hidden patterns that are shared by the tasks. In deep learning, a significant challenge in the backpropagation training process is the design of advanced optimisers to improve the convergence speed and stability of the gradient descent learning rule. In particular, in multi-task deep learning (MTDL) the multitude of tasks may generate potentially conflicting gradients that would hinder the concurrent convergence of the diverse loss functions. This challenge arises when the gradients of the task objectives have either different magnitudes or opposite directions, causing one or a few to dominate or to interfere with each other, thus degrading the training process. Gradient surgery methods address the problem explicitly dealing with conflicting gradients by adjusting the overall gradient trajectory. This work introduces a novel gradient surgery method, the Similarity-Aware Momentum Gradient Surgery (SAM-GS), which provides an effective and scalable approach based on a gradient magnitude similarity measure to guide the optimisation process. The SAM-GS surgery adopts gradient equalisation and modulation of the first-order momentum. A series of experimental tests have shown the effectiveness of SAM-GS on synthetic problems and MTL benchmarks. Gradient magnitude similarity plays a crucial role in regularising gradient aggregation in MTDL for the optimisation of the learning process.