mlsanityData to 14 Oct 2026

Do Current Multi-Task Optimization Methods in Deep Learning Even Help?

arXiv 2209.11379 ↗First posted 23 Sep 2022cs.LG cs.AIcited by 60

Not on the map. Why?

Abstract (arXiv)

Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply optimizing a weighted average of the task losses. In this paper, we perform large-scale experiments on a variety of language and vision tasks to examine the empirical validity of these claims. We show that, despite the added design and computational complexity of these algorithms, MTO methods do not yield any performance improvements beyond what is achievable via traditional optimization approaches. We highlight alternative strategies that consistently yield improvements to the performance profile and point out common training pitfalls that might cause suboptimal results. Finally, we outline challenges in reliably evaluating the performance of MTO algorithms and discuss potential solutions.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 23 Sep 2022.

Builds on

23 of 40 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  5. 2021Convex optimizationcited by 2,515
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  10. 2020In Search of Lost Domain Generalizationcited by 1,054
  11. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
11 more
  1. 2015LSTM: A Search Space Odysseycited by 316
  2. 2017Automated Curriculum Learning for Neural Networkscited by 257
  3. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  4. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  5. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  6. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  7. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  8. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  9. 2021Unsupervised Domain Adaptation: A Reality Checkcited by 24
  10. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  11. 2021Robust Optimization for Multilingual Translation with Imbalanced Datacited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 19; 2024: 20; 2025: 13; 2026: 5.

60 papers.

Sort:

By area:

  1. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
  2. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
  3. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  4. 2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 36
  5. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
  6. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 31
  7. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  8. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  9. 2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
  10. 2023Dual-Balancing for Multi-Task Learningcited by 27
  11. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  12. 2023Expressive Losses for Verified Robustness via Convex Combinationscited by 23
38 more
  1. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  2. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  3. 2024Jacobian Descent for Multi-Objective Optimizationcited by 18
  4. 2024Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimizationcited by 16
  5. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
  6. 2023Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendationcited by 13
  7. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  8. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  9. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  10. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  11. 2023On the Pareto Front of Multilingual Neural Machine Translationcited by 7
  12. 2023Efficient Controllable Multi-Task Architecturescited by 7
  13. 2023Order Matters in the Presence of Dataset Imbalance for Multilingual Learningcited by 7
  14. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  15. 2022Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?cited by 6
  16. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  17. 2024Joint-Task Regularization for Partially Labeled Multi-Task Learningcited by 6
  18. 2024Analytical Uncertainty-Based Loss Weighting in Multi-Task Learningcited by 6
  19. 2024Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasetscited by 6
  20. 2024Multitask Learning in Minimally Invasive Surgical Vision: A Reviewcited by 5
  21. 2023Differentiable Random Partition Modelscited by 3
  22. 2024Cross-Task Affinity Learning for Multitask Dense Scene Predictionscited by 3
  23. 2024Federated Communication-Efficient Multi-Objective Optimizationcited by 3
  24. 2025Continual Optimization with Symmetry Teleportation for Multi-Task Learningcited by 3
  25. 2025Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysiscited by 3
  26. 2023Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignmentcited by 2
  27. 2023ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scalecited by 2
  28. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
  29. 2024Can Optimization Trajectories Explain Multi-Task Transfer?cited by 2
  30. 2025DeepChest: Dynamic Gradient-Free Task Weighting for Effective Multi-Task Learning in Chest X-ray Classificationcited by 2
  31. 2025Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learningcited by 2
  32. 2025AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metricscited by 2
  33. 2024Aux-NAS: Exploiting Auxiliary Labels with Negligibly Extra Inference Costcited by 1
  34. 2024Label Augmentation for Neural Networks Robustnesscited by 1
  35. 2025Characterization of Transfer Using Multi-task Learning Curvescited by 1
  36. 2026Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimizationcited by 1
  37. 2026End-to-End Fairness Optimization with Fair Decision-Focused Learningcited by 1
  38. 2023Exploring Representational Disparities Between Multilingual and Bilingual Translation Modelscited by 0

50 most cited of 60 All 60 (JSON)