# Robust Multi-Task Learning with Excess Risks

arXiv: https://arxiv.org/abs/2402.02009 · HTML: https://mlsanity.com/paper/2402.02009/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [NLNL: Negative Learning for Noisy Labels](https://mlsanity.com/paper/1908.07387/) (2019)
- [Instance-aware Semantic Segmentation via Multi-task Network Cascades](https://mlsanity.com/paper/1512.04412/) (2015)
- [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](https://mlsanity.com/paper/2111.10603/) (2021)
- [Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective](https://mlsanity.com/paper/2308.13985/) (2023)
- [Hard-Aware Deeply Cascaded Embedding](https://mlsanity.com/paper/1611.05720/) (2016)
- [Multi-Objective Meta Learning](https://mlsanity.com/paper/2102.07121/) (2021)
- [The Complexity of Finding Stationary Points with Stochastic Gradient Descent](https://mlsanity.com/paper/1910.01845/) (2019)
- [Focus on the Common Good: Group Distributional Robustness Follows](https://mlsanity.com/paper/2110.02619/) (2021)
- [LibMTL: A Python Library for Multi-Task Learning](https://mlsanity.com/paper/2203.14338/) (2022)
- [On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification](https://mlsanity.com/paper/2207.13975/) (2022)
- [Balancing Average and Worst-case Accuracy in Multitask Learning](https://mlsanity.com/paper/2110.05838/) (2021)
- [Hard Sample Mining for the Improved Retraining of Automatic Speech Recognition](https://mlsanity.com/paper/1904.08031/) (2019)

## Built on by (most cited first, 24 of 24)

- [Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic](https://mlsanity.com/paper/2408.13656/) (2024)
- [Scaling Laws for Multilingual Language Models](https://mlsanity.com/paper/2410.12883/) (2024)
- [Model Predictive Task Sampling for Efficient and Robust Adaptation](https://mlsanity.com/paper/2501.11039/) (2025)
- [Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimization](https://mlsanity.com/paper/2405.19650/) (2024)
- [Test-Time Alignment via Hypothesis Reweighting](https://mlsanity.com/paper/2412.08812/) (2024)
- [MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning](https://mlsanity.com/paper/2505.24846/) (2025)
- [LibMOON: A Gradient-based MultiObjective OptimizatioN Library in PyTorch](https://mlsanity.com/paper/2409.02969/) (2024)
- [AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs](https://mlsanity.com/paper/2506.00569/) (2025)
- [Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery](https://mlsanity.com/paper/2510.22124/) (2025)
- [Adaptive Online Mirror Descent for Tchebycheff Scalarization in Multi-Objective Learning](https://mlsanity.com/paper/2410.21764/) (2024)
- [LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis](https://mlsanity.com/paper/2510.24561/) (2025)
- [On the Plasticity and Stability for Post-Training Large Language Models](https://mlsanity.com/paper/2602.06453/) (2026)
- [Aligned Multi Objective Optimization](https://mlsanity.com/paper/2502.14096/) (2025)
- [AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics](https://mlsanity.com/paper/2508.13979/) (2025)
- [Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution](https://mlsanity.com/paper/2505.21438/) (2025)
- [Graph Coloring for Multi-Task Learning](https://mlsanity.com/paper/2509.16959/) (2025)
- [ERGO: Excess-Risk-Guided Optimization for High-Fidelity Monocular 3D Gaussian Splatting](https://mlsanity.com/paper/2602.10278/) (2026)
- [MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning](https://mlsanity.com/paper/2608.11749/) (2026)
- [Quality-Diversity Optimization as Multi-Objective Optimization](https://mlsanity.com/paper/2602.00478/) (2026)
- [From Dual Tracking to Clipping: Provably Faster Distributionally Robust Multi-Objective Optimization](https://mlsanity.com/paper/2605.05660/) (2026)
- [Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling](https://mlsanity.com/paper/2605.14350/) (2026)
- [CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representations](https://mlsanity.com/paper/2606.02221/) (2026)
- [MAdam: Metric-Aware Multi-Objective Adam](https://mlsanity.com/paper/2606.03904/) (2026)
- [DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression](https://mlsanity.com/paper/2610.19206/) (2026)

## Abstract (arXiv)

Multi-task learning (MTL) considers learning a joint model for multiple tasks by optimizing a convex combination of all task losses. To solve the optimization problem, existing methods use an adaptive weight updating scheme, where task weights are dynamically adjusted based on their respective losses to prioritize difficult tasks. However, these algorithms face a great challenge whenever label noise is present, in which case excessive weights tend to be assigned to noisy tasks that have relatively large Bayes optimal errors, thereby overshadowing other tasks and causing performance to drop across the board. To overcome this limitation, we propose Multi-Task Learning with Excess Risks (ExcessMTL), an excess risk-based task balancing method that updates the task weights by their distances to convergence instead. Intuitively, ExcessMTL assigns higher weights to worse-trained tasks that are further from convergence. To estimate the excess risks, we develop an efficient and accurate method with Taylor approximation. Theoretically, we show that our proposed algorithm achieves convergence guarantees and Pareto stationarity. Empirically, we evaluate our algorithm on various MTL benchmarks and demonstrate its superior performance over existing methods in the presence of label noise. Our code is available at https://github.com/yifei-he/ExcessMTL.