# Mixing ADAM and SGD: a Combined Optimization Method

arXiv: https://arxiv.org/abs/2011.08042 · HTML: https://mlsanity.com/paper/2011.08042/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 8 of 8)

- [Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models](https://mlsanity.com/paper/2410.06441/) (2024)
- [New Insight in Cervical Cancer Diagnosis Using Convolution Neural Network Architecture](https://mlsanity.com/paper/2410.17735/) (2024)
- [Weber-Fechner Law in Temporal Difference learning derived from Control as Inference](https://mlsanity.com/paper/2412.21004/) (2024)
- [Break a Lag: Triple Exponential Moving Average for Enhanced Optimization](https://mlsanity.com/paper/2306.01423/) (2023)
- [Domain adaption and physical constrains transfer learning for shale gas production](https://mlsanity.com/paper/2312.10920/) (2023)
- [Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning](https://mlsanity.com/paper/2408.15421/) (2024)
- [FUSE: First-Order and Second-Order Unified SynthEsis in Stochastic Optimization](https://mlsanity.com/paper/2503.04204/) (2025)
- [Efficient training for large-scale optical neural network using an evolutionary strategy and attention pruning](https://mlsanity.com/paper/2505.12906/) (2025)

## Abstract (arXiv)

Optimization methods (optimizers) get special attention for the efficient training of neural networks in the field of deep learning. In literature there are many papers that compare neural models trained with the use of different optimizers. Each paper demonstrates that for a particular problem an optimizer is better than the others but as the problem changes this type of result is no longer valid and we have to start from scratch. In our paper we propose to use the combination of two very different optimizers but when used simultaneously they can overcome the performances of the single optimizers in very different problems. We propose a new optimizer called MAS (Mixing ADAM and SGD) that integrates SGD and ADAM simultaneously by weighing the contributions of both through the assignment of constant weights. Rather than trying to improve SGD or ADAM we exploit both at the same time by taking the best of both. We have conducted several experiments on images and text document classification, using various CNNs, and we demonstrated by experiments that the proposed MAS optimizer produces better performance than the single SGD or ADAM optimizers. The source code and all the results of the experiments are available online at the following link https://gitlab.com/nicolalandro/multioptimizer