# Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective

arXiv: https://arxiv.org/abs/2308.13985 · HTML: https://mlsanity.com/paper/2308.13985/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 27 Aug 2023.

## Builds on (most cited first, 26 of 26)

- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Learning the Pareto Front with Hypernetworks](https://mlsanity.com/paper/2010.04104/) (2020)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](https://mlsanity.com/paper/2111.10603/) (2021)
- [In Defense of the Unitary Scalarization for Deep Multi-Task Learning](https://mlsanity.com/paper/2201.04122/) (2022)
- [Do Current Multi-Task Optimization Methods in Deep Learning Even Help?](https://mlsanity.com/paper/2209.11379/) (2022)
- [Efficient Continuous Pareto Exploration in Multi-Task Learning](https://mlsanity.com/paper/2006.16434/) (2020)
- [Controllable Pareto Multi-Task Learning](https://mlsanity.com/paper/2010.06313/) (2020)
- [Fair and Optimal Classification via Post-Processing](https://mlsanity.com/paper/2211.01528/) (2022)
- [Scalable Pareto Front Approximation for Deep Multi-Objective Learning](https://mlsanity.com/paper/2103.13392/) (2021)
- [Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance](https://mlsanity.com/paper/2305.20057/) (2023)
- [Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Set](https://mlsanity.com/paper/2110.08713/) (2021)

## Built on by (most cited first, 30 of 51)

- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://mlsanity.com/paper/2402.18571/) (2024)
- [Representation Surgery for Multi-Task Model Merging](https://mlsanity.com/paper/2402.02705/) (2024)
- [Panacea: Pareto Alignment via Preference Adaptation for LLMs](https://mlsanity.com/paper/2402.02030/) (2024)
- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic](https://mlsanity.com/paper/2408.13656/) (2024)
- [Fair Resource Allocation in Multi-Task Learning](https://mlsanity.com/paper/2402.15638/) (2024)
- [MergeBench: A Benchmark for Merging Domain-Specialized LLMs](https://mlsanity.com/paper/2505.10833/) (2025)
- [Robust Multi-Task Learning with Excess Risks](https://mlsanity.com/paper/2402.02009/) (2024)
- [CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging](https://mlsanity.com/paper/2505.06977/) (2025)
- [Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models](https://mlsanity.com/paper/2507.01915/) (2025)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimization](https://mlsanity.com/paper/2405.19650/) (2024)
- [Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2407.17466/) (2024)
- [All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation](https://mlsanity.com/paper/2409.19660/) (2024)
- [Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts](https://mlsanity.com/paper/2501.15065/) (2025)
- [Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey](https://mlsanity.com/paper/2403.01909/) (2024)
- [SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery](https://mlsanity.com/paper/2410.14389/) (2024)
- [LibMOON: A Gradient-based MultiObjective OptimizatioN Library in PyTorch](https://mlsanity.com/paper/2409.02969/) (2024)
- [Generative or Discriminative? Revisiting Text Classification in the Era of Transformers](https://mlsanity.com/paper/2506.12181/) (2025)
- [TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2508.04682/) (2025)
- [Enhancing Domain Adaptation through Prompt Gradient Alignment](https://mlsanity.com/paper/2406.09353/) (2024)
- [Fairness-Aware Meta-Learning via Nash Bargaining](https://mlsanity.com/paper/2406.07029/) (2024)
- [Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards](https://mlsanity.com/paper/2510.01167/) (2025)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Balanced Rate-Distortion Optimization in Learned Image Compression](https://mlsanity.com/paper/2502.20161/) (2025)
- [Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery](https://mlsanity.com/paper/2510.22124/) (2025)
- [Adaptive Online Mirror Descent for Tchebycheff Scalarization in Multi-Objective Learning](https://mlsanity.com/paper/2410.21764/) (2024)
- [Z-Erase: Enabling Concept Erasure in Single-Stream Diffusion Transformers](https://mlsanity.com/paper/2603.25074/) (2026)
- [Federated Communication-Efficient Multi-Objective Optimization](https://mlsanity.com/paper/2410.16398/) (2024)

## Abstract (arXiv)

Linear scalarization, i.e., combining all loss functions by a weighted sum, has been the default choice in the literature of multi-task learning (MTL) since its inception. In recent years, there is a surge of interest in developing Specialized Multi-Task Optimizers (SMTOs) that treat MTL as a multi-objective optimization problem. However, it remains open whether there is a fundamental advantage of SMTOs over scalarization. In fact, heated debates exist in the community comparing these two types of algorithms, mostly from an empirical perspective. To approach the above question, in this paper, we revisit scalarization from a theoretical perspective. We focus on linear MTL models and study whether scalarization is capable of fully exploring the Pareto front. Our findings reveal that, in contrast to recent works that claimed empirical advantages of scalarization, scalarization is inherently incapable of full exploration, especially for those Pareto optimal solutions that strike the balanced trade-offs between multiple tasks. More concretely, when the model is under-parametrized, we reveal a multi-surface structure of the feasible region and identify necessary and sufficient conditions for full exploration. This leads to the conclusion that scalarization is in general incapable of tracing out the Pareto front. Our theoretical results partially answer the open questions in Xin et al. (2021), and provide a more intuitive explanation on why scalarization fails beyond non-convexity. We additionally perform experiments on a real-world dataset using both scalarization and state-of-the-art SMTOs. The experimental results not only corroborate our theoretical findings, but also unveil the potential of SMTOs in finding balanced solutions, which cannot be achieved by scalarization.