mlsanityData to 7 Oct 2026

In Defense of the Unitary Scalarization for Deep Multi-Task Learning

arXiv 2201.04122 ↗First posted 11 Jan 2022cs.LG cs.AI cs.CVcited by 65

Not on the map. Why?

Abstract (arXiv)

Recent multi-task learning research argues against unitary scalarization, where training simply minimizes the sum of the task losses. Several ad-hoc multi-task optimization algorithms have instead been proposed, inspired by various hypotheses about what makes multi-task settings difficult. The majority of these optimizers require per-task gradients, and introduce significant memory, runtime, and implementation overhead. We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings. We then present an analysis suggesting that many specialized multi-task optimizers can be partly interpreted as forms of regularization, potentially explaining our surprising results. We believe our results call for a critical reevaluation of recent research in the area.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 44 of its 77 references that are in the map. First posted on arXiv 11 Jan 2022.

Builds on

44 of 77 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  9. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
32 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  4. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2017Dynamic Routing Between Capsulescited by 689
  7. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  8. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  9. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  10. 2015Policy Distillationcited by 414
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  12. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  13. 2017Dilated Residual Networkscited by 293
  14. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  15. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
  16. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  17. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  18. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  19. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  20. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  21. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  22. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  23. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  24. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  25. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  26. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  27. 2020Learning to Branch for Multi-Task Learningcited by 114
  28. 2020One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Controlcited by 114
  29. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
  30. 2016Learning values across many orders of magnitudecited by 80
  31. 2020My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Controlcited by 68
  32. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 24; 2024: 14; 2025: 11; 2026: 3.

65 papers.

Sort:

By area:

  1. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  2. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 74
  3. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  4. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  6. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  7. 2022A Generalist Neural Algorithmic Learnercited by 58
  8. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  9. 2023From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Predictioncited by 39
  10. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  11. 2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 35
  12. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
38 more
  1. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  2. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  3. 2023Dual-Balancing for Multi-Task Learningcited by 27
  4. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  5. 2023Expressive Losses for Verified Robustness via Convex Combinationscited by 23
  6. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  7. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  8. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 18
  9. 2024Jacobian Descent for Multi-Objective Optimizationcited by 18
  10. 2022Hybrid Multi-agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systemscited by 17
  11. 2024Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimizationcited by 15
  12. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  13. 2023Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendationcited by 13
  14. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 12
  15. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  16. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  17. 2023Improvable Gap Balancing for Multi-Task Learningcited by 9
  18. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learningcited by 9
  19. 2022Beyond Transfer Learning: Co-finetuning for Action Localisationcited by 8
  20. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  21. 2025Multi-Task Reinforcement Learning Enables Parameter Scalingcited by 8
  22. 2023Efficient Controllable Multi-Task Architecturescited by 7
  23. 2023Order Matters in the Presence of Dataset Imbalance for Multilingual Learningcited by 7
  24. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  25. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  26. 2023Multi-Modal Multi-Task (3MT) Road Segmentationcited by 6
  27. 2024MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalitiescited by 6
  28. 2024Analytical Uncertainty-Based Loss Weighting in Multi-Task Learningcited by 6
  29. 2024Vertical Federated Learning with Missing Features During Training and Inferencecited by 6
  30. 2024Multitask Learning in Minimally Invasive Surgical Vision: A Reviewcited by 5
  31. 2025One Step is Enough: Multi-Agent Reinforcement Learning based on One-Step Policy Optimization for Order Dispatch on Ride-Sharing Platformscited by 5
  32. 2022Vertical Layering of Quantized Neural Networks for Heterogeneous Inferencecited by 4
  33. 2023Differentiable Random Partition Modelscited by 3
  34. 2024Federated Communication-Efficient Multi-Objective Optimizationcited by 3
  35. 2025Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysiscited by 3
  36. 2023FairBranch: Mitigating Bias Transfer in Fair Multi-task Learningcited by 2
  37. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
  38. 2024Can Optimization Trajectories Explain Multi-Task Transfer?cited by 2

50 most cited of 65 All 65 (JSON)