# MTAdam: Automatic Balancing of Multiple Training Loss Terms

arXiv: https://arxiv.org/abs/2006.14683 · HTML: https://mlsanity.com/paper/2006.14683/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 13 of 13)

- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks](https://mlsanity.com/paper/2203.06801/) (2022)
- [Dual-Balancing for Multi-Task Learning](https://mlsanity.com/paper/2308.12029/) (2023)
- [Adversarial sampling of unknown and high-dimensional conditional distributions](https://mlsanity.com/paper/2111.05962/) (2021)
- [Real-time Monocular Depth Estimation with Sparse Supervision on Mobile](https://mlsanity.com/paper/2105.12053/) (2021)
- [Multi-behavior Self-supervised Learning for Recommendation](https://mlsanity.com/paper/2305.18238/) (2023)
- [Client-specific Property Inference against Secure Aggregation in Federated Learning](https://mlsanity.com/paper/2303.03908/) (2023)
- [Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval](https://mlsanity.com/paper/2204.13382/) (2022)
- [MultiBalance: Multi-Objective Gradient Balancing in Industrial-Scale Multi-Task Recommendation System](https://mlsanity.com/paper/2411.11871/) (2024)
- [Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning](https://mlsanity.com/paper/2507.21049/) (2025)
- [On Training Derivative-Constrained Neural Networks](https://mlsanity.com/paper/2310.01649/) (2023)
- [Learning Generalized Hamiltonian Dynamics with Stability from Noisy Trajectory Data](https://mlsanity.com/paper/2509.07280/) (2025)
- [MAdam: Metric-Aware Multi-Objective Adam](https://mlsanity.com/paper/2606.03904/) (2026)

## Abstract (arXiv)

When training neural models, it is common to combine multiple loss terms. The balancing of these terms requires considerable human effort and is computationally demanding. Moreover, the optimal trade-off between the loss term can change as training progresses, especially for adversarial terms. In this work, we generalize the Adam optimization algorithm to handle multiple loss terms. The guiding principle is that for every layer, the gradient magnitude of the terms should be balanced. To this end, the Multi-Term Adam (MTAdam) computes the derivative of each loss term separately, infers the first and second moments per parameter and loss term, and calculates a first moment for the magnitude per layer of the gradients arising from each loss. This magnitude is used to continuously balance the gradients across all layers, in a manner that both varies from one layer to the next and dynamically changes over time. Our results show that training with the new method leads to fast recovery from suboptimal initial loss weighting and to training outcomes that match conventional training with the prescribed hyperparameters of each method.