mlsanityData to 14 Oct 2026

Efficiently Identifying Task Groupings for Multi-Task Learning

arXiv 2109.04617 ↗First posted 10 Sep 2021cs.LG cs.AI cs.CVcited by 222

Not on the map. Why?

Abstract (arXiv)

Multi-task learning can leverage information learned by one task to benefit the training of other tasks. Despite this capacity, naively training all tasks together in one model often degrades performance, and exhaustively searching through combinations of task groupings can be prohibitively expensive. As a result, efficiently identifying the tasks that would benefit from training together remains a challenging design question without a clear solution. In this paper, we suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single run by training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss. On the large-scale Taskonomy computer vision dataset, we find this method can decrease test loss by 10.0% compared to simply training all tasks together while operating 11.6 times faster than a state-of-the-art task grouping method.

Key facts. Cited by 222 papers in the map (data through 14 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

39 of 62 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2016Progressive Neural Networkscited by 1,361
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  10. 2017A Survey on Multi-Task Learningcited by 840
  11. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
27 more
  1. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  2. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  3. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  4. 2017Progressive Neural Architecture Searchcited by 487
  5. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  6. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  7. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  9. 2019Pareto Multi-Task Learningcited by 224
  10. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  11. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  12. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  13. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  14. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  15. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 160
  16. 2020Learning to Branch for Multi-Task Learningcited by 115
  17. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 114
  18. 2016Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classificationcited by 93
  19. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  20. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  21. 2016Trace Norm Regularised Deep Multi-Task Learningcited by 50
  22. 2018A Modulation Module for Multi-task Learning with Applications in Image Retrievalcited by 49
  23. 2019The Information Complexity of Learning Tasks, their Structure and their Distancecited by 36
  24. 2019Regularizing Deep Multi-Task Networks using Orthogonal Gradientscited by 36
  25. 2018GNAS: A Greedy Neural Architecture Search Method for Multi-Attribute Learningcited by 17
  26. 2017Deep Asymmetric Multi-task Feature Learningcited by 15
  27. 2019Chameleon: Learning Model Initializations Across Tasks With Different Schemascited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 29; 2023: 57; 2024: 52; 2025: 47; 2026: 33.

222 papers.

Sort:

By area:

  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  3. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 352
  4. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  5. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 156
  6. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  7. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  8. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  9. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  10. 2023Deep Model Fusion: A Surveycited by 91
  11. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
  12. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
38 more
  1. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  2. 2024Parameter Competition Balancing for Model Mergingcited by 72
  3. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  4. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  5. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 60
  6. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  7. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  8. 2022Multitask Vision-Language Prompt Tuningcited by 51
  9. 2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 51
  10. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  11. 2023MmAP : Multi-modal Alignment Prompt for Cross-domain Multi-task Learningcited by 41
  12. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  13. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  14. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 34
  15. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
  16. 2024Training-Free Pretrained Model Mergingcited by 32
  17. 2022Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Drivingcited by 30
  18. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  19. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  20. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  21. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  22. 2023Denoising Task Routing for Diffusion Modelscited by 28
  23. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28
  24. 2024Denoising Task Difficulty-based Curriculum for Training Diffusion Modelscited by 27
  25. 2022Wasserstein Task Embedding for Measuring Task Similaritiescited by 25
  26. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 23
  27. 2023Identification of Negative Transfers in Multitask Learning Using Surrogate Modelscited by 20
  28. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  29. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  30. 2024Jacobian Descent for Multi-Objective Optimizationcited by 18
  31. 2021AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learningcited by 17
  32. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 17
  33. 2024Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimizationcited by 16
  34. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  35. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  36. 2023Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flightcited by 15
  37. 2022Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approachcited by 14
  38. 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14

50 most cited of 222 All 222 (JSON)