# MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks

arXiv: https://arxiv.org/abs/2203.06801 · HTML: https://mlsanity.com/paper/2203.06801/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [Auxiliary Tasks in Multi-task Learning](https://mlsanity.com/paper/1805.06334/) (2018)
- [Adapting Auxiliary Losses Using Gradient Similarity](https://mlsanity.com/paper/1812.02224/) (2018)
- [Self-Supervised Generalisation with Meta Auxiliary Learning](https://mlsanity.com/paper/1901.08933/) (2019)
- [Deep Auxiliary Learning for Visual Localization and Odometry](https://mlsanity.com/paper/1803.03642/) (2018)
- [Learning Longer-term Dependencies in RNNs with Auxiliary Losses](https://mlsanity.com/paper/1803.00144/) (2018)
- [Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition](https://mlsanity.com/paper/1704.01631/) (2017)
- [Ask the GRU: Multi-Task Learning for Deep Text Recommendations](https://mlsanity.com/paper/1609.02116/) (2016)
- [MTAdam: Automatic Balancing of Multiple Training Loss Terms](https://mlsanity.com/paper/2006.14683/) (2020)
- [A Hierarchical Self-Attentive Model for Recommending User-Generated Item Lists](https://mlsanity.com/paper/1912.13023/) (2019)

## Built on by (most cited first, 30 of 34)

- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Representation Surgery for Multi-Task Model Merging](https://mlsanity.com/paper/2402.02705/) (2024)
- [MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance](https://mlsanity.com/paper/2405.17730/) (2024)
- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic](https://mlsanity.com/paper/2406.11385/) (2024)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework](https://mlsanity.com/paper/2404.18465/) (2024)
- [AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning](https://mlsanity.com/paper/2211.15055/) (2022)
- [Dual-Balancing for Multi-Task Learning](https://mlsanity.com/paper/2308.12029/) (2023)
- [CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging](https://mlsanity.com/paper/2505.06977/) (2025)
- [STEM: Unleashing the Power of Embeddings for Multi-task Recommendation](https://mlsanity.com/paper/2308.13537/) (2023)
- [Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts](https://mlsanity.com/paper/2501.15065/) (2025)
- [SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery](https://mlsanity.com/paper/2410.14389/) (2024)
- [Multi-behavior Self-supervised Learning for Recommendation](https://mlsanity.com/paper/2305.18238/) (2023)
- [ChangeMinds: Multi-task Framework for Detecting and Describing Changes in Remote Sensing](https://mlsanity.com/paper/2410.10047/) (2024)
- [Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings](https://mlsanity.com/paper/2506.07109/) (2025)
- [Sample-Level Weighting for Multi-Task Learning with Auxiliary Tasks](https://mlsanity.com/paper/2306.04519/) (2023)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [Careful with that Scalpel: Improving Gradient Surgery with an EMA](https://mlsanity.com/paper/2402.02998/) (2024)
- [Socialized Learning: A Survey of the Paradigm Shift for Edge Intelligence in Networked Systems](https://mlsanity.com/paper/2404.13348/) (2024)
- [Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning](https://mlsanity.com/paper/2507.21049/) (2025)
- [Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning](https://mlsanity.com/paper/2303.05847/) (2023)
- [UPNet: Uncertainty-based Picking Deep Learning Network for Robust First Break Picking](https://mlsanity.com/paper/2305.13799/) (2023)
- [Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptation](https://mlsanity.com/paper/2401.16299/) (2024)
- [Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition](https://mlsanity.com/paper/2509.20840/) (2025)
- [Locating and Mitigating Gradient Conflicts in Point Cloud Domain Adaptation via Saliency Map Skewness](https://mlsanity.com/paper/2504.15796/) (2025)
- [Personalized Product Search Ranking: A Multi-Task Learning Approach with Tabular and Non-Tabular Data](https://mlsanity.com/paper/2508.09636/) (2025)
- [Momentum-integrated Multi-task Stock Recommendation with Converge-based Optimization](https://mlsanity.com/paper/2509.10461/) (2025)
- [Uncertainty Modeling for Multi-Objective RTA Interception with Distillation Acceleration](https://mlsanity.com/paper/2511.05582/) (2025)
- [Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimization](https://mlsanity.com/paper/2603.07389/) (2026)

## Abstract (arXiv)

In many personalized recommendation scenarios, the generalization ability of a target task can be improved via learning with additional auxiliary tasks alongside this target task on a multi-task network. However, this method often suffers from a serious optimization imbalance problem. On the one hand, one or more auxiliary tasks might have a larger influence than the target task and even dominate the network weights, resulting in worse recommendation accuracy for the target task. On the other hand, the influence of one or more auxiliary tasks might be too weak to assist the target task. More challenging is that this imbalance dynamically changes throughout the training process and varies across the parts of the same network. We propose a new method: MetaBalance to balance auxiliary losses via directly manipulating their gradients w.r.t the shared parameters in the multi-task network. Specifically, in each training iteration and adaptively for each part of the network, the gradient of an auxiliary loss is carefully reduced or enlarged to have a closer magnitude to the gradient of the target loss, preventing auxiliary tasks from being so strong that dominate the target task or too weak to help the target task. Moreover, the proximity between the gradient magnitudes can be flexibly adjusted to adapt MetaBalance to different scenarios. The experiments show that our proposed method achieves a significant improvement of 8.34% in terms of NDCG@10 upon the strongest baseline on two real-world datasets. The code of our approach can be found at here: https://github.com/facebookresearch/MetaBalance