mlsanityData to 14 Oct 2026

ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning

arXiv 2301.12618 ↗First posted 30 Jan 2023cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Auxiliary-Task Learning (ATL) aims to improve the performance of the target task by leveraging the knowledge obtained from related tasks. Occasionally, learning multiple tasks simultaneously results in lower accuracy than learning only the target task, which is known as negative transfer. This problem is often attributed to the gradient conflicts among tasks, and is frequently tackled by coordinating the task gradients in previous works. However, these optimization-based methods largely overlook the auxiliary-target generalization capability. To better understand the root cause of negative transfer, we experimentally investigate it from both optimization and generalization perspectives. Based on our findings, we introduce ForkMerge, a novel approach that periodically forks the model into multiple branches, automatically searches the varying task weights by minimizing target validation errors, and dynamically merges all branches to filter out detrimental task-parameter updates. On a series of auxiliary-task learning benchmarks, ForkMerge outperforms existing methods and effectively mitigates negative transfer.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 60 of its 87 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

60 of 87 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Mask R-CNNcited by 7,277
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  10. 2016Wide Residual Networkscited by 3,070
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
38 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  3. 2016Progressive Neural Networkscited by 1,361
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  7. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  10. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  11. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  13. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  14. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  15. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  16. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  17. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  18. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  19. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  20. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  21. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  22. 2018Characterizing and Avoiding Negative Transfercited by 229
  23. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  24. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  25. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  26. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  27. 2014Qualitatively characterizing neural network optimization problemscited by 188
  28. 2018Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Ratecited by 173
  29. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  30. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  31. 2020A Survey on Negative Transfercited by 122
  32. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  33. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 100
  34. 2018Importance Weighted Adversarial Nets for Partial Domain Adaptationcited by 90
  35. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  36. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  37. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  38. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 14; 2026: 2.

28 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  3. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  4. 2023Deep Model Fusion: A Surveycited by 91
  5. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  6. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 17
  7. 2024StainDiffuser: MultiTask Dual Diffusion Model for Virtual Stainingcited by 11
  8. 2025HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Drivingcited by 6
  9. 2025FW-Merging: Scaling Model Merging with Frank-Wolfe Optimizationcited by 5
  10. 2024ADSNet: Cross-Domain LTV Prediction with an Adaptive Siamese Network in Advertisingcited by 4
  11. 2025Large Language Model as Universal Retriever in Industrial-Scale Recommender Systemcited by 4
  12. 2025Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learningcited by 4
16 more
  1. 2024Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuningcited by 3
  2. 2025Pareto Continual Learning: Preference-Conditioned Learning and Adaption for Dynamic Stability-Plasticity Trade-offcited by 3
  3. 2024Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learningcited by 2
  4. 2024Can Optimization Trajectories Explain Multi-Task Transfer?cited by 2
  5. 2024TTVD: Towards a Geometric Framework for Test-Time Adaptation Based on Voronoi Diagramcited by 2
  6. 2024From Basic to Extra Features: Hypergraph Transformer Pretrain-then-Finetuning for Balanced Clinical Predictions on EHRcited by 1
  7. 2024Variational Model Merging for Pareto Front Estimation in Multitask Finetuningcited by 1
  8. 2025Graceful Forgetting in Generative Language Modelscited by 1
  9. 2025Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Datacited by 1
  10. 2025Mitigating Negative Transfer via Reducing Environmental Disagreementcited by 1
  11. 2025Auxiliary Gene Learning: Spatial Gene Expression Estimation by Auxiliary Gene Selectioncited by 1
  12. 2025Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transportcited by 1
  13. 2025A Graph Neural Network with Auxiliary Task Learning for Missing PMU Data Reconstructioncited by 0
  14. 2025Semi-overlapping Multi-bandit Best Arm Identification for Sequential Support Network Learningcited by 0
  15. 2026MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Expertscited by 0
  16. 2026Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imagingcited by 0