mlsanityData to 14 Oct 2026

Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning

arXiv 2111.10603 ↗First posted 20 Nov 2021cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

Multi-Task Learning (MTL) has achieved success in various fields. However, how to balance different tasks to achieve good performance is a key problem. To achieve the task balancing, there are many works to carefully design dynamical loss/gradient weighting strategies but the basic random experiments are ignored to examine their effectiveness. In this paper, we propose the Random Weighting (RW) methods, including Random Loss Weighting (RLW) and Random Gradient Weighting (RGW), where an MTL model is trained with random loss/gradient weights sampled from a distribution. To show the effectiveness and necessity of RW methods, theoretically we analyze the convergence of RW and reveal that RW has a higher probability to escape local minima, resulting in better generalization ability. Empirically, we extensively evaluate the proposed RW methods to compare with twelve state-of-the-art methods on five image datasets and two multilingual problems from the XTREME benchmark to show RW methods can achieve comparable performance with state-of-the-art baselines. Therefore, we think that the RW methods are important baselines for MTL and should attract more attentions.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 20 Nov 2021.

Builds on

39 of 51 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  12. 2017A Survey on Multi-Task Learningcited by 840
27 more
  1. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  2. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  4. 2017Dynamic Routing Between Capsulescited by 689
  5. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  6. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  8. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  9. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  11. 2017Dilated Residual Networkscited by 294
  12. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  13. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  14. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  15. 2019Pareto Multi-Task Learningcited by 224
  16. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  17. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  18. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  19. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  20. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  21. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 133
  22. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  23. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  24. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  25. 2021Multi-Objective Meta Learningcited by 39
  26. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  27. 2019A Principled Approach for Learning Task Similarity in Multitask Learningcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 34; 2025: 23; 2026: 20.

101 papers.

Sort:

By area:

  1. 2023Independent Component Alignment for Multi-Task Learningcited by 70
  2. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  3. 2024ConFIG: Towards Conflict-free Training of Physics Informed Neural Networkscited by 46
  4. 2023Shape-Erased Feature Learning for Visible-Infrared Person Re-Identificationcited by 38
  5. 2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 36
  6. 2024Fair Resource Allocation in Multi-Task Learningcited by 35
  7. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 34
  8. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 31
  9. 2024CosmicMan: A Text-to-Image Foundation Model for Humanscited by 30
  10. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  11. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  12. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 28
38 more
  1. 2023Dual-Balancing for Multi-Task Learningcited by 27
  2. 2023Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyondcited by 26
  3. 2024Robust Multi-Task Learning with Excess Riskscited by 24
  4. 2024An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Modelscited by 23
  5. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
  6. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  7. 2023Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitivescited by 18
  8. 2024Jacobian Descent for Multi-Objective Optimizationcited by 18
  9. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  10. 2024Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimizationcited by 16
  11. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  12. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  13. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  14. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  15. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  16. 2025MTL-UE: Learning to Learn Nothing for Multi-Task Learningcited by 11
  17. 2024MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoderscited by 10
  18. 2024LibMOON: A Gradient-based MultiObjective OptimizatioN Library in PyTorchcited by 10
  19. 2023Improvable Gap Balancing for Multi-Task Learningcited by 9
  20. 2025TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Predictioncited by 9
  21. 2023Hi-ResNet: Edge Detail Enhancement for High-Resolution Remote Sensing Segmentationcited by 8
  22. 2024A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimizationcited by 8
  23. 2024MGDA Converges under Generalized Smoothness, Provablycited by 7
  24. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  25. 2023Sample-Level Weighting for Multi-Task Learning with Auxiliary Taskscited by 6
  26. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  27. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  28. 2024Analytical Uncertainty-Based Loss Weighting in Multi-Task Learningcited by 6
  29. 2023Improving End-to-End SLU performance with Prosodic Attention and Distillationcited by 5
  30. 2023BYOM: Building Your Own Multi-Task Model For Freecited by 5
  31. 2024See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRIcited by 5
  32. 2025Gradient Similarity Surgery in Multi-Task Deep Learningcited by 5
  33. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  34. 2024Adaptive Task Balancing for Visual Instruction Tuning via Inter-Task Contribution and Intra-Task Difficultycited by 4
  35. 2024Using dynamic loss weighting to boost improvements in forecast stabilitycited by 4
  36. 2025FairHuman: Boosting Hand and Face Quality in Human Image Generation with Minimum Potential Delay Fairness in Diffusion Modelscited by 4
  37. 2024Multi-Task Learning with Multi-Task Optimizationcited by 3
  38. 2025Aligned Multi Objective Optimizationcited by 3

50 most cited of 101 All 101 (JSON)