mlsanityData to 13 Oct 2026

RotoGrad: Gradient Homogenization in Multitask Learning

arXiv 2103.02631 ↗First posted 3 Mar 2021cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Multitask learning is being increasingly adopted in applications domains like computer vision and reinforcement learning. However, optimally exploiting its advantages remains a major challenge due to the effect of negative transfer. Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks, when optimizing the shared network parameters. While recent work has acknowledged that negative transfer is a two-fold problem, existing approaches fall short as they only focus on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conflicts. In this work, we introduce RotoGrad, an algorithm that tackles negative transfer as a whole: it jointly homogenizes gradient magnitudes and directions, while ensuring training convergence. We show that RotoGrad outperforms competing methods in complex problems, including multi-label classification in CelebA and computer vision tasks in the NYUv2 dataset. A Pytorch implementation can be found in https://github.com/adrianjav/rotograd.

Key facts. Cited by 82 papers in the map (data through 13 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 3 Mar 2021.

Builds on

28 of 45 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,686
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,882
  6. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,484
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,348
  8. 2020Gradient Surgery for Multi-Task Learningcited by 1,225
  9. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 822
  10. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 812
  11. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 799
  12. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 769
16 more
  1. 2018Taskonomy: Disentangling Task Transfer Learningcited by 633
  2. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  3. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  4. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 368
  5. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  6. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  7. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  8. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  9. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  10. 2020Learning to Branch for Multi-Task Learningcited by 115
  11. 2019Meta-Learning with Warped Gradient Descentcited by 98
  12. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  13. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  14. 2013Indoor Semantic Segmentation using depth informationcited by 56
  15. 2017Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Pointscited by 29
  16. 2018Gradient Adversarial Training of Neural Networkscited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 22; 2024: 19; 2025: 18; 2026: 12.

82 papers.

Sort:

By area:

  1. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  2. 2023Modular Deep Learningcited by 83
  3. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  4. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  5. 2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 50
  6. 2024MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learningcited by 44
  7. 2024ConFIG: Towards Conflict-free Training of Physics Informed Neural Networkscited by 44
  8. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 33
  9. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
  10. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 30
  11. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  12. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
38 more
  1. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  2. 2023FedHCA2: Towards Hetero-Client Federated Multi-Task Learningcited by 17
  3. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 16
  4. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  5. 2023Symmetry-Aware Robot Design with Structured Subgroupscited by 13
  6. 2023Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendationcited by 13
  7. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  8. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 13
  9. 2023Auxiliary Learning as an Asymmetric Bargaining Gamecited by 12
  10. 2025Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflictscited by 12
  11. 2026DanceOPD: On-Policy Generative Field Distillationcited by 12
  12. 2024Quantifying Task Priority for Multi-Task Optimizationcited by 11
  13. 2023MulGT: Multi-task Graph-Transformer with Task-aware Knowledge Injection and Domain Knowledge-driven Pooling for Whole Slide Image Analysiscited by 9
  14. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  15. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  16. 2023Roman Numeral Analysis with Graph Neural Networks: Onset-wise Predictions from Note-wise Featurescited by 7
  17. 2023Multi-domain Recommendation with Embedding Disentangling and Domain Alignmentcited by 7
  18. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 7
  19. 2025Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalizationcited by 7
  20. 2022Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?cited by 6
  21. 2023Sample-Level Weighting for Multi-Task Learning with Auxiliary Taskscited by 6
  22. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  23. 2024Analytical Uncertainty-Based Loss Weighting in Multi-Task Learningcited by 6
  24. 2025HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Drivingcited by 6
  25. 2023RotoGBML: Towards Out-of-Distribution Generalization for Gradient-Based Meta-Learningcited by 5
  26. 2023Multi-task View Synthesis with Neural Radiance Fieldscited by 5
  27. 2024PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimizationcited by 5
  28. 2025Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgerycited by 5
  29. 2022CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrationscited by 4
  30. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
  31. 2025Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learningcited by 4
  32. 2023Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNscited by 3
  33. 2023OFVL-MS: Once for Visual Localization across Multiple Indoor Scenescited by 3
  34. 2024Towards Principled Task Grouping for Multi-Task Learningcited by 3
  35. 2024Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradientscited by 3
  36. 2025Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysiscited by 3
  37. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
  38. 2024Can Optimization Trajectories Explain Multi-Task Transfer?cited by 2

50 most cited of 82 All 82 (JSON)