mlsanityData to 7 Oct 2026

Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

arXiv 2010.06808 ↗First posted 14 Oct 2020cs.LG cs.CVcited by 199

Not on the map. Why?

Abstract (arXiv)

The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these multiple updates can impede optimal training by pulling the model in conflicting directions. We present Gradient Sign Dropout (GradDrop), a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. GradDrop is implemented as a simple deep layer that can be used in any deep net and synergizes with other gradient balancing approaches. We show that GradDrop outperforms the state-of-the-art multiloss methods within traditional multitask and transfer learning settings, and we discuss how GradDrop reveals links between optimal multiloss training and gradient stochasticity.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 26; 2023: 34; 2024: 53; 2025: 47; 2026: 25.

199 papers.

Sort:

By area:

  1. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  2. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  3. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  4. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  5. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  6. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  7. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  8. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  9. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  10. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  11. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  12. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
38 more
  1. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 74
  2. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  3. 2023Independent Component Alignment for Multi-Task Learningcited by 69
  4. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  5. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  6. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  7. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  8. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  9. 2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 50
  10. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 50
  11. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  12. 2024ConFIG: Towards Conflict-free Training of Physics Informed Neural Networkscited by 43
  13. 2021Learning Multiple Dense Prediction Tasks from Partially Annotated Datacited by 42
  14. 2021Understanding and Improving Fairness-Accuracy Trade-offs in Multi-Task Learningcited by 34
  15. 2022MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Taskscited by 34
  16. 2024Fair Resource Allocation in Multi-Task Learningcited by 34
  17. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 33
  18. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
  19. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  20. 2022AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learningcited by 31
  21. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  22. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  23. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  24. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  25. 2023Denoising Task Routing for Diffusion Modelscited by 28
  26. 2023Dual-Balancing for Multi-Task Learningcited by 27
  27. 2022LibMTL: A Python Library for Multi-Task Learningcited by 25
  28. 2022Universal Representations: A Unified Look at Multiple Task and Domain Learningcited by 23
  29. 2024Robust Multi-Task Learning with Excess Riskscited by 23
  30. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  31. 2023InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understandingcited by 22
  32. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 22
  33. 2022Controllable Dynamic Multi-Task Architecturescited by 21
  34. 2023STEM: Unleashing the Power of Embeddings for Multi-task Recommendationcited by 19
  35. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  36. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  37. 2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
  38. 2024Jacobian Descent for Multi-Objective Optimizationcited by 18

50 most cited of 199 All 199 (JSON)