# AutoLoss: Learning Discrete Schedules for Alternate Optimization

arXiv: https://arxiv.org/abs/1810.02442 · HTML: https://mlsanity.com/paper/1810.02442/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2018.

## Built on by (most cited first, 12 of 12)

- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Automated Self-Supervised Learning for Graphs](https://mlsanity.com/paper/2106.05470/) (2021)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)
- [T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging](https://mlsanity.com/paper/2111.07535/) (2021)
- [Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.04935/) (2021)
- [AutoLossGen: Automatic Loss Function Generation for Recommender Systems](https://mlsanity.com/paper/2204.13160/) (2022)
- [CCMN: A General Framework for Learning with Class-Conditional Multi-Label Noise](https://mlsanity.com/paper/2105.07338/) (2021)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [Improve Noise Tolerance of Robust Loss via Noise-Awareness](https://mlsanity.com/paper/2301.07306/) (2023)
- [Label Budget Allocation in Multi-Task Learning](https://mlsanity.com/paper/2308.12949/) (2023)
- [DLC: A Metric-Guided Dynamic Loss Controller for Multi-Objective Training](https://mlsanity.com/paper/2610.11433/) (2026)

## Abstract (arXiv)

Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -- it can guide and improve the learning of a new task model with different specifications, or on different datasets.