mlsanityData to 14 Oct 2026

Optimizer Amalgamation

arXiv 2203.06474 ↗First posted 12 Mar 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners. Many analytical optimizers have been proposed using a variety of theoretical and empirical approaches; however, none can offer a universal advantage over other competitive optimizers. We are thus motivated to study a new problem named Optimizer Amalgamation: how can we best combine a pool of "teacher" optimizers into a single "student" optimizer that can have stronger problem-specific performance? In this paper, we draw inspiration from the field of "learning to optimize" to use a learnable amalgamation target. First, we define three differentiable amalgamation mechanisms to amalgamate a pool of analytical optimizers by gradient descent. Then, in order to reduce variance of the amalgamation process, we also explore methods to stabilize the amalgamation process by perturbing the amalgamation target. Finally, we present experiments showing the superiority of our amalgamated optimizer compared to its amalgamated components and learning to optimize baselines, and the efficacy of our variance reducing perturbations. Our code and pre-trained models are publicly available at http://github.com/VITA-Group/OptimizerAmalgamation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 12 Mar 2022.

Builds on

46 of 70 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  9. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  10. 2016Deep Learning with Differential Privacycited by 2,728
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
34 more
  1. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  2. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  4. 2016Learning to learn by gradient descent by gradient descentcited by 620
  5. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  6. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  7. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  8. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  9. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  10. 2021Learning to Optimize: A Primer and A Benchmarkcited by 208
  11. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  12. 2017Non-convex Optimization for Machine Learningcited by 133
  13. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 125
  14. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  15. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  16. 2017Learned Optimizers that Scale and Generalizecited by 110
  17. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  18. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  19. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  20. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  21. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  22. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  23. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  24. 2020L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networkscited by 48
  25. 2020Self-PU: Self Boosted and Calibrated Positive-Unlabeled Trainingcited by 47
  26. 2018Amalgamating Knowledge towards Comprehensive Classificationcited by 38
  27. 2020Automated Synthetic-to-Real Generalizationcited by 35
  28. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  29. 2020Training Stronger Baselines for Learning to Optimizecited by 31
  30. 2019Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and Morecited by 29
  31. 2019Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learningcited by 28
  32. 2019Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamationcited by 23
  33. 2019Learning to Optimize in Swarmscited by 23
  34. 2020Improved Adversarial Training via Learned Optimizercited by 22

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgettingcited by 0