# ATE-SG: Alternate Through the Epochs Stochastic Gradient for Multi-Task Neural Networks

arXiv: https://arxiv.org/abs/2312.16340 · HTML: https://mlsanity.com/paper/2312.16340/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [Many Task Learning with Task Routing](https://mlsanity.com/paper/1903.12117/) (2019)
- [The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning](https://mlsanity.com/paper/1907.04472/) (2019)
- [Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels](https://mlsanity.com/paper/1908.09597/) (2019)
- [Accuracy and Fairness Trade-offs in Machine Learning: A Stochastic Multi-Objective Approach](https://mlsanity.com/paper/2008.01132/) (2020)
- [Block stochastic gradient iteration for convex and nonconvex optimization](https://mlsanity.com/paper/1408.2597/) (2014)
- [Multi-task Learning Approach for Automatic Modulation and Wireless Signal Classification](https://mlsanity.com/paper/2101.10254/) (2021)
- [Improved optimization strategies for deep Multi-Task Networks](https://mlsanity.com/paper/2109.11678/) (2021)
- [Convergence rates of the stochastic alternating algorithm for bi-objective optimization](https://mlsanity.com/paper/2203.10605/) (2022)

## Abstract (arXiv)

This paper introduces novel alternate training procedures for hard-parameter sharing Multi-Task Neural Networks (MTNNs). Traditional MTNN training faces challenges in managing conflicting loss gradients, often yielding sub-optimal performance. The proposed alternate training method updates shared and task-specific weights alternately through the epochs, exploiting the multi-head architecture of the model. This approach reduces computational costs per epoch and memory requirements. Convergence properties similar to those of the classical stochastic gradient method are established. Empirical experiments demonstrate enhanced training regularization and reduced computational demands. In summary, our alternate training procedures offer a promising advancement for the training of hard-parameter sharing MTNNs.