# Identification of Negative Transfers in Multitask Learning Using Surrogate Models

arXiv: https://arxiv.org/abs/2303.14582 · HTML: https://mlsanity.com/paper/2303.14582/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 25 Mar 2023.

## Builds on (most cited first, 30 of 38)

- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [Representer Point Selection for Explaining Deep Neural Networks](https://mlsanity.com/paper/1811.09720/) (2018)
- [Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations](https://mlsanity.com/paper/2203.01517/) (2022)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [Training Complex Models with Multi-Task Weak Supervision](https://mlsanity.com/paper/1810.02840/) (2018)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Auto-Lambda: Disentangling Dynamic Task Relationships](https://mlsanity.com/paper/2202.03091/) (2022)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [On the Value of Target Data in Transfer Learning](https://mlsanity.com/paper/2002.04747/) (2020)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Improved Regularization and Robustness for Fine-tuning in Neural Networks](https://mlsanity.com/paper/2111.04578/) (2021)
- [Meta-learning for mixed linear regression](https://mlsanity.com/paper/2002.08936/) (2020)
- [Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion](https://mlsanity.com/paper/2302.04451/) (2023)

## Built on by (most cited first, 20 of 20)

- [Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity](https://mlsanity.com/paper/2409.06091/) (2024)
- [Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit](https://mlsanity.com/paper/2409.17717/) (2024)
- [Boosting Multitask Learning on Graphs through Higher-Order Task Affinities](https://mlsanity.com/paper/2306.14009/) (2023)
- [Loss-to-Loss Prediction: Scaling Laws for All Datasets](https://mlsanity.com/paper/2411.12925/) (2024)
- [Graph Neural Networks for Road Safety Modeling: Datasets and Evaluations for Accident Analysis](https://mlsanity.com/paper/2311.00164/) (2023)
- [U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection](https://mlsanity.com/paper/2501.09687/) (2025)
- [Multi-group Learning for Hierarchical Groups](https://mlsanity.com/paper/2402.00258/) (2024)
- [Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation](https://mlsanity.com/paper/2511.12779/) (2025)
- [Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information](https://mlsanity.com/paper/2410.12774/) (2024)
- [Efficient Estimation of Kernel Surrogate Models for Task Attribution](https://mlsanity.com/paper/2602.03783/) (2026)
- [Can Optimization Trajectories Explain Multi-Task Transfer?](https://mlsanity.com/paper/2408.14677/) (2024)
- [Learning Tree-Structured Composition of Data Augmentation](https://mlsanity.com/paper/2408.14381/) (2024)
- [Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution](https://mlsanity.com/paper/2505.21438/) (2025)
- [Efficiently Learning Branching Networks for Multitask Algorithmic Reasoning](https://mlsanity.com/paper/2512.01113/) (2025)
- [Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries](https://mlsanity.com/paper/2410.15875/) (2024)
- [Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimation](https://mlsanity.com/paper/2512.02920/) (2025)
- [Ensemble Prediction of Task Affinity for Efficient Multi-Task Learning](https://mlsanity.com/paper/2602.18591/) (2026)
- [Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimization](https://mlsanity.com/paper/2603.07389/) (2026)
- [B3-Net: Controlled Posterior Bridge Learning for Multi-Task Dense Prediction](https://mlsanity.com/paper/2605.05722/) (2026)
- [A helps B while B hurts A: directed transfer in instruction-tuning mixture](https://mlsanity.com/paper/2609.39702/) (2026)

## Abstract (arXiv)

Multitask learning is widely used in practice to train a low-resource target task by augmenting it with multiple related source tasks. Yet, naively combining all the source tasks with a target task does not always improve the prediction performance for the target task due to negative transfers. Thus, a critical problem in multitask learning is identifying subsets of source tasks that would benefit the target task. This problem is computationally challenging since the number of subsets grows exponentially with the number of source tasks; efficient heuristics for subset selection do not always capture the relationship between task subsets and multitask learning performances. In this paper, we introduce an efficient procedure to address this problem via surrogate modeling. In surrogate modeling, we sample (random) subsets of source tasks and precompute their multitask learning performances. Then, we approximate the precomputed performances with a linear regression model that can also predict the multitask performance of unseen task subsets. We show theoretically and empirically that fitting this model only requires sampling linearly many subsets in the number of source tasks. The fitted model provides a relevance score between each source and target task. We use the relevance scores to perform subset selection for multitask learning by thresholding. Through extensive experiments, we show that our approach predicts negative transfers from multiple source tasks to target tasks much more accurately than existing task affinity measures. Additionally, we demonstrate that for several weak supervision datasets, our approach consistently improves upon existing optimization methods for multitask learning.