mlsanityData to 14 Oct 2026

Quantifying Task Priority for Multi-Task Optimization

arXiv 2406.02996 ↗First posted 5 Jun 2024cs.LG cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

The goal of multi-task learning is to learn diverse tasks within a single unified network. As each task has its own unique objective function, conflicts emerge during training, resulting in negative transfer among them. Earlier research identified these conflicting gradients in shared parameters between tasks and attempted to realign them in the same direction. However, we prove that such optimization strategies lead to sub-optimal Pareto solutions due to their inability to accurately determine the individual contributions of each parameter across various tasks. In this paper, we propose the concept of task priority to evaluate parameter contributions across different tasks. To learn task priority, we identify the type of connections related to links between parameters influenced by task-specific losses during backpropagation. The strength of connections is gauged by the magnitude of parameters to determine task priority. Based on these, we present a new method named connection strength-based optimization for multi-task learning which consists of two phases. The first phase learns the task priority within the network, while the second phase modifies the gradients while upholding this priority. This ultimately leads to finding new Pareto optimal solutions for multiple tasks. Through extensive experiments, we show that our approach greatly enhances multi-task performance in comparison to earlier gradient manipulation methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

37 of 50 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  7. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  9. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  12. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
25 more
  1. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  2. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  3. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  4. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  5. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  6. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  7. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  8. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  9. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  10. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  11. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  12. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  13. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  14. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  15. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  16. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 142
  17. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  18. 2015Instance-aware Semantic Segmentation via Multi-task Network Cascadescited by 121
  19. 2014Deep Convolutional Neural Fields for Depth Estimation from a Single Imagecited by 106
  20. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  21. 2017Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimationcited by 70
  22. 2023Independent Component Alignment for Multi-Task Learningcited by 70
  23. 2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 51
  24. 2016Convolutional Neural Fabricscited by 32
  25. 2020Filter Sketch for Network Pruningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 4.

11 papers.

  1. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 17
  2. 2025Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planningcited by 5
  3. 2025Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Trainingcited by 5
  4. 2024radarODE-MTL: A Multi-Task Learning Framework with Eccentric Gradient Alignment for Robust Radar-Based ECG Reconstructioncited by 4
  5. 2025Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learningcited by 4
  6. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  7. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  8. 2025Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Predictioncited by 1
  9. 2025SAR2Struct: Extracting 3D Semantic Structural Representation of Aircraft Targets from Single-View SAR Imagecited by 0
  10. 2026The Double Dilemma in Multi-Task Radiology Report Generation: A Gradient Dynamics Analysis and Solutioncited by 0
  11. 2026Triggering Generalist Reasoning via Predictive Uncertainty for Dual-System VLAcited by 0