mlsanityData to 7 Oct 2026

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

arXiv 1711.02257 ↗First posted 7 Nov 2017cs.CVcited by 816

Not on the map. Why?

Cited by 22 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep multitask networks, in which one neural network produces multiple predictive outputs, can offer better speed and performance than their single-task counterparts but are challenging to train properly. We present a gradient normalization (GradNorm) algorithm that automatically balances training in deep multitask models by dynamically tuning gradient magnitudes. We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting across multiple tasks when compared to single-task networks, static baselines, and other adaptive multitask loss balancing techniques. GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter α. Thus, what was once a tedious search process that incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks. Ultimately, we will demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

Key facts. Cited by 816 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 96; 2022: 91; 2023: 121; 2024: 140; 2025: 170; 2026: 198.

816 papers.

Sort:

By area:

  1. 2021ImageNet-21K Pretraining for the Massescited by 607
  2. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  3. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  4. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  5. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  6. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  8. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  9. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  10. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  11. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  12. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
38 more
  1. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  2. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  3. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  4. 2021Geometry Uncertainty Projection Network for Monocular 3D Object Detectioncited by 155
  5. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  6. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 128
  7. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  8. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  9. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  10. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  11. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  12. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  13. 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 86
  14. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  15. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
  16. 2021Cross-domain Few-shot Learning with Task-specific Adapterscited by 79
  17. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 74
  18. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  19. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  20. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  21. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  22. 2023Independent Component Alignment for Multi-Task Learningcited by 69
  23. 20223D Object Detection from Images for Autonomous Driving: A Surveycited by 66
  24. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  25. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  26. 2022CLIPascene: Scene Sketching with Different Types and Levels of Abstractioncited by 63
  27. 2021Domain Generalization via Gradient Surgerycited by 60
  28. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  29. 2024Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyondcited by 60
  30. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  31. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  32. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  33. 2021OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Drivingcited by 57
  34. 2021Universal Representation Learning from Multiple Domains for Few-shot Classificationcited by 56
  35. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  36. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  37. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  38. 2021Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learningcited by 50

50 most cited of 816 All 816 (JSON)