# Dual-Balancing for Multi-Task Learning

arXiv: https://arxiv.org/abs/2308.12029 · HTML: https://mlsanity.com/paper/2308.12029/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 70 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](https://mlsanity.com/paper/2106.03760/) (2021)

## Built on by (most cited first, 27 of 27)

- [A Survey of Reasoning with Foundation Models](https://mlsanity.com/paper/2312.11562/) (2023)
- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model](https://mlsanity.com/paper/2505.06274/) (2025)
- [MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders](https://mlsanity.com/paper/2407.02228/) (2024)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders](https://mlsanity.com/paper/2408.15101/) (2024)
- [A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimization](https://mlsanity.com/paper/2401.09257/) (2024)
- [Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning](https://mlsanity.com/paper/2408.07985/) (2024)
- [BYOM: Building Your Own Multi-Task Model For Free](https://mlsanity.com/paper/2310.01886/) (2023)
- [UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment](https://mlsanity.com/paper/2602.09538/) (2026)
- [ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both](https://mlsanity.com/paper/2605.15198/) (2026)
- [Examining Common Paradigms in Multi-Task Learning](https://mlsanity.com/paper/2311.04698/) (2023)
- [MultiBalance: Multi-Objective Gradient Balancing in Industrial-Scale Multi-Task Recommendation System](https://mlsanity.com/paper/2411.11871/) (2024)
- [Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification](https://mlsanity.com/paper/2502.07299/) (2025)
- [Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning](https://mlsanity.com/paper/2507.21049/) (2025)
- [AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics](https://mlsanity.com/paper/2508.13979/) (2025)
- [NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2510.18258/) (2025)
- [Uncovering Cross-Objective Interference in Multi-Objective Alignment](https://mlsanity.com/paper/2602.06869/) (2026)
- [Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution](https://mlsanity.com/paper/2505.21438/) (2025)
- [EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery](https://mlsanity.com/paper/2506.06830/) (2025)
- [Semimage: HSV-Based Semantic Image Encoding for Disentangled Text Representation](https://mlsanity.com/paper/2512.00088/) (2025)
- [Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning](https://mlsanity.com/paper/2505.10347/) (2025)
- [FastCAR: Fast Classification And Regression for Task Consolidation in Multi-Task Learning to Model a Continuous Property Variable of Detected Object Class](https://mlsanity.com/paper/2506.00208/) (2025)
- [Momentum-integrated Multi-task Stock Recommendation with Converge-based Optimization](https://mlsanity.com/paper/2509.10461/) (2025)
- [CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representations](https://mlsanity.com/paper/2606.02221/) (2026)
- [MAdam: Metric-Aware Multi-Objective Adam](https://mlsanity.com/paper/2606.03904/) (2026)
- [A Shared-Backbone Approach for Multi-Task MedMNIST Classification](https://mlsanity.com/paper/2609.06838/) (2026)

## Abstract (arXiv)

Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art.