# Scalarization for Multi-Task and Multi-Domain Learning at Scale

arXiv: https://arxiv.org/abs/2310.08910 · HTML: https://mlsanity.com/paper/2310.08910/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 66 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 30 of 48)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [Incremental Learning Through Deep Adaptation](https://mlsanity.com/paper/1705.04228/) (2017)

## Built on by (most cited first, 19 of 19)

- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity](https://mlsanity.com/paper/2409.06091/) (2024)
- [Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimization](https://mlsanity.com/paper/2405.19650/) (2024)
- [AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers](https://mlsanity.com/paper/2411.11188/) (2024)
- [MTL-UE: Learning to Learn Nothing for Multi-Task Learning](https://mlsanity.com/paper/2505.05279/) (2025)
- [Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning](https://mlsanity.com/paper/2501.15556/) (2025)
- [DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method](https://mlsanity.com/paper/2508.17054/) (2025)
- [Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets](https://mlsanity.com/paper/2410.04579/) (2024)
- [Nighttime Person Re-Identification via Collaborative Enhancement Network with Multi-domain Learning](https://mlsanity.com/paper/2312.16246/) (2023)
- [Federated Communication-Efficient Multi-Objective Optimization](https://mlsanity.com/paper/2410.16398/) (2024)
- [Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis](https://mlsanity.com/paper/2509.23915/) (2025)
- [Examining Common Paradigms in Multi-Task Learning](https://mlsanity.com/paper/2311.04698/) (2023)
- [Can Optimization Trajectories Explain Multi-Task Transfer?](https://mlsanity.com/paper/2408.14677/) (2024)
- [AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics](https://mlsanity.com/paper/2508.13979/) (2025)
- [InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning](https://mlsanity.com/paper/2402.16848/) (2024)
- [Aux-NAS: Exploiting Auxiliary Labels with Negligibly Extra Inference Cost](https://mlsanity.com/paper/2405.05695/) (2024)
- [Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning](https://mlsanity.com/paper/2511.07198/) (2025)
- [Dynamic Domain Information Modulation Algorithm for Multi-domain Sentiment Analysis](https://mlsanity.com/paper/2505.06630/) (2025)
- [Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning](https://mlsanity.com/paper/2505.10347/) (2025)

## Abstract (arXiv)

Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone hence improves model efficiency. It also enables potential positive knowledge transfer across tasks/domains, leading to improved accuracy and data-efficient training. However, optimizing such networks is a challenge, in particular due to discrepancies between the different tasks or domains: Despite several hypotheses and solutions proposed over the years, recent work has shown that uniform scalarization training, i.e., simply minimizing the average of the task losses, yields on-par performance with more costly SotA optimization methods. This raises the issue of how well we understand the training dynamics of multi-task and multi-domain networks. In this work, we first devise a large-scale unified analysis of multi-domain and multi-task learning to better understand the dynamics of scalarization across varied task/domain combinations and model sizes. Following these insights, we then propose to leverage population-based training to efficiently search for the optimal scalarization weights when dealing with a large number of tasks or domains.