# Mitigating Negative Transfer in Multi-Task Learning with Exponential Moving Average Loss Weighting Strategies

arXiv: https://arxiv.org/abs/2211.12999 · HTML: https://mlsanity.com/paper/2211.12999/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 11 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique](https://mlsanity.com/paper/2107.04192/) (2021)

## Built on by (most cited first, 10 of 10)

- [Joint-Task Regularization for Partially Labeled Multi-Task Learning](https://mlsanity.com/paper/2404.01976/) (2024)
- [Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning](https://mlsanity.com/paper/2408.07985/) (2024)
- [Multi-Task Learning with Multi-Annotation Triplet Loss for Improved Object Detection](https://mlsanity.com/paper/2504.08054/) (2025)
- [Deep multitask neural networks for solving some stochastic optimal control problems](https://mlsanity.com/paper/2401.12923/) (2024)
- [Semi Supervised Heterogeneous Domain Adaptation via Disentanglement and Pseudo-Labelling](https://mlsanity.com/paper/2406.14087/) (2024)
- [DINOv3 as a Frozen Encoder for CRPS-Oriented Probabilistic Rainfall Nowcasting](https://mlsanity.com/paper/2511.10894/) (2025)
- [Multi-Kernel Gated Decoder Adapters for Robust Multi-Task Thyroid Ultrasound under Cross-Center Shift](https://mlsanity.com/paper/2603.08906/) (2026)
- [SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning](https://mlsanity.com/paper/2609.12599/) (2026)
- [Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging](https://mlsanity.com/paper/2609.18688/) (2026)
- [Distillation for Efficient Multitask Manipulation Policies via Conditional Flow Matching](https://mlsanity.com/paper/2609.28107/) (2026)

## Abstract (arXiv)

Multi-Task Learning (MTL) is a growing subject of interest in deep learning, due to its ability to train models more efficiently on multiple tasks compared to using a group of conventional single-task models. However, MTL can be impractical as certain tasks can dominate training and hurt performance in others, thus making some tasks perform better in a single-task model compared to a multi-task one. Such problems are broadly classified as negative transfer, and many prior approaches in the literature have been made to mitigate these issues. One such current approach to alleviate negative transfer is to weight each of the losses so that they are on the same scale. Whereas current loss balancing approaches rely on either optimization or complex numerical analysis, none directly scale the losses based on their observed magnitudes. We propose multiple techniques for loss balancing based on scaling by the exponential moving average and benchmark them against current best-performing methods on three established datasets. On these datasets, they achieve comparable, if not higher, performance compared to current best-performing methods.