mlsanityData to 14 Oct 2026

Understanding and Improving Information Transfer in Multi-Task Learning

arXiv 2005.00944 ↗First posted 2 May 2020cs.LG cs.CLcited by 114

Not on the map. Why?

Abstract (arXiv)

We investigate multi-task learning approaches that use a shared feature representation for all tasks. To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task. We study the theory of this setting on linear and ReLU-activated models. Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtain a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT-LARGE using our alignment method. We also design an SVD-based task reweighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 26; 2023: 29; 2024: 16; 2025: 16; 2026: 9.

114 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  3. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  4. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  5. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  6. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  7. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  8. 2022Universal Prompt Tuning for Graph Neural Networkscited by 100
  9. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  10. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  11. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  12. 2023LoRA ensembles for large language model fine-tuningcited by 67
38 more
  1. 2024Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyondcited by 60
  2. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  3. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  4. 2023MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphscited by 40
  5. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  6. 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
  7. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  8. 2021Understanding and Improving Fairness-Accuracy Trade-offs in Multi-Task Learningcited by 34
  9. 2022Adaptive and Robust Multi-Task Learningcited by 34
  10. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  11. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
  12. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  13. 2022Fair Generative Models via Transfer Learningcited by 27
  14. 2023Identification of Negative Transfers in Multitask Learning Using Surrogate Modelscited by 20
  15. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  16. 2025Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMscited by 19
  17. 2022Control-Aware Prediction Objectives for Autonomous Drivingcited by 18
  18. 2023Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitivescited by 18
  19. 2023Estimating Treatment Effects from Irregular Time Series Observations with Hidden Confounderscited by 17
  20. 2023A Universal Semantic-Geometric Representation for Robotic Manipulationcited by 16
  21. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  22. 2024Condition-Aware Neural Network for Controlled Image Generationcited by 16
  23. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  24. 2022Provable and Efficient Continual Representation Learningcited by 14
  25. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  26. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
  27. 2023Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Datacited by 13
  28. 2023Graph Neural Networks for Road Safety Modeling: Datasets and Evaluations for Accident Analysiscited by 13
  29. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  30. 2021Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Setcited by 12
  31. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  32. 2021On the Complementarity of Data Selection and Fine Tuning for Domain Adaptationcited by 10
  33. 2023Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learningcited by 10
  34. 2025Efficient Ensemble for Fine-tuning Language Models on Multiple Datasetscited by 10
  35. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  36. 2023Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learningcited by 9
  37. 2024A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectivescited by 9
  38. 2024Diffusion Models For Multi-Modal Generative Modelingcited by 9

50 most cited of 114 All 114 (JSON)