mlsanityData to 14 Oct 2026

Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning

arXiv 2505.10347 ↗First posted 15 May 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Specialized Multi-Task Optimizers (SMTOs) balance task learning in Multi-Task Learning by addressing issues like conflicting gradients and differing gradient norms, which hinder equal-weighted task training. However, recent critiques suggest that equally weighted tasks can achieve competitive results compared to SMTOs, arguing that previous SMTO results were influenced by poor hyperparameter optimization and lack of regularization. In this work, we evaluate these claims through an extensive empirical evaluation of SMTOs, including some of the latest methods, on more complex multi-task problems to clarify this behavior. Our findings indicate that SMTOs perform well compared to uniform loss and that fixed weights can achieve competitive performance compared to SMTOs. Furthermore, we demonstrate why uniform loss perform similarly to SMTOs in some instances. The source code is available at https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

39 of 50 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  12. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
27 more
  1. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  2. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  3. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  4. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  5. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  6. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  7. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  8. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  9. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  10. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  11. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  12. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  13. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 114
  14. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  15. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  16. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
  17. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  18. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  19. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  20. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  21. 2023Dual-Balancing for Multi-Task Learningcited by 27
  22. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  23. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  24. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  25. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  26. 2020Follow the bisector: a simple method for multi-objective optimizationcited by 4
  27. 2022Leveraging convergence behavior to balance conflicting tasks in multi-task learningcited by 2

Built on by

Not cited yet.