mlsanityData to 14 Oct 2026

Dual-Balancing for Multi-Task Learning

arXiv 2308.12029 ↗First posted 23 Aug 2023cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

46 of 70 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  7. 2015Fast R-CNNcited by 2,933
  8. 2021Convex optimizationcited by 2,515
  9. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  10. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  11. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  12. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
34 more
  1. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  2. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  3. 2017A Survey on Multi-Task Learningcited by 840
  4. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  5. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  6. 2020Directional Message Passing for Molecular Graphscited by 561
  7. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  8. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  9. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  10. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  11. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  12. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  13. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  14. 2018Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Ratecited by 173
  15. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  16. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  17. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  18. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  19. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  20. 2017Adversarial Multi-task Learning for Text Classificationcited by 91
  21. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  22. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  23. 2023Independent Component Alignment for Multi-Task Learningcited by 70
  24. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  25. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 62
  26. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  27. 2019MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learningcited by 45
  28. 2021Multi-Objective Meta Learningcited by 39
  29. 2022MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Taskscited by 34
  30. 2019Learning Sparse Sharing Architectures for Multiple Taskscited by 31
  31. 2023Multi-Task Deep Recommender Systems: A Surveycited by 30
  32. 2022LibMTL: A Python Library for Multi-Task Learningcited by 25
  33. 2020MTAdam: Automatic Balancing of Multiple Training Loss Termscited by 13
  34. 2023Improvable Gap Balancing for Multi-Task Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 12; 2026: 6.

27 papers.

Sort:

By area:

  1. 2023A Survey of Reasoning with Foundation Modelscited by 70
  2. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
  3. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 28
  4. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
  5. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  6. 2024MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoderscited by 10
  7. 2024A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimizationcited by 8
  8. 2024Analytical Uncertainty-Based Loss Weighting in Multi-Task Learningcited by 6
  9. 2023BYOM: Building Your Own Multi-Task Model For Freecited by 5
  10. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  11. 2026ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Bothcited by 3
  12. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
15 more
  1. 2024MultiBalance: Multi-Objective Gradient Balancing in Industrial-Scale Multi-Task Recommendation Systemcited by 2
  2. 2025Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unificationcited by 2
  3. 2025Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learningcited by 2
  4. 2025AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metricscited by 2
  5. 2025NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspectivecited by 2
  6. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignmentcited by 2
  7. 2025Measuring Fine-Grained Relatedness in Multitask Learning via Data Attributioncited by 1
  8. 2025EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgerycited by 1
  9. 2025Semimage: HSV-Based Semantic Image Encoding for Disentangled Text Representationcited by 1
  10. 2025Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learningcited by 0
  11. 2025FastCAR: Fast Classification And Regression for Task Consolidation in Multi-Task Learning to Model a Continuous Property Variable of Detected Object Classcited by 0
  12. 2025Momentum-integrated Multi-task Stock Recommendation with Converge-based Optimizationcited by 0
  13. 2026CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representationscited by 0
  14. 2026MAdam: Metric-Aware Multi-Objective Adamcited by 0
  15. 2026A Shared-Backbone Approach for Multi-Task MedMNIST Classificationcited by 0