# A Survey of Optimization Methods from a Machine Learning Perspective

arXiv: https://arxiv.org/abs/1906.06821 · HTML: https://mlsanity.com/paper/1906.06821/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2019.

## Built on by (most cited first, 30 of 91)

- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [Re-parameterizing Your Optimizers rather than Architectures](https://mlsanity.com/paper/2205.15242/) (2022)
- [Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers](https://mlsanity.com/paper/2402.17564/) (2024)
- [Optimization with Constraint Learning: A Framework and Survey](https://mlsanity.com/paper/2110.02121/) (2021)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimization](https://mlsanity.com/paper/2411.00625/) (2024)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search](https://mlsanity.com/paper/2102.11646/) (2021)
- [VectorAdam for Rotation Equivariant Geometry Optimization](https://mlsanity.com/paper/2205.13599/) (2022)
- [Prεεmpt: Sanitizing Sensitive Prompts for LLMs](https://mlsanity.com/paper/2504.05147/) (2025)
- [AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization](https://mlsanity.com/paper/2201.06714/) (2022)
- [Generative Adversarial Model-Based Optimization via Source Critic Regularization](https://mlsanity.com/paper/2402.06532/) (2024)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Recommender systems based on graph embedding techniques: A comprehensive review](https://mlsanity.com/paper/2109.09587/) (2021)
- [A Gaussian Process Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2401.03492/) (2024)
- [Comparative analysis of machine learning methods for active flow control](https://mlsanity.com/paper/2202.11664/) (2022)
- [Unsupervised Tissue Segmentation via Deep Constrained Gaussian Network](https://mlsanity.com/paper/2208.02912/) (2022)
- [Using Human Perception to Regularize Transfer Learning](https://mlsanity.com/paper/2211.07885/) (2022)
- [A Continuous Convolutional Trainable Filter for Modelling Unstructured Data](https://mlsanity.com/paper/2210.13416/) (2022)
- [Quantum-enhanced Computer Vision: Going Beyond Classical Algorithms](https://mlsanity.com/paper/2510.07317/) (2025)
- [Machine Learning (ML)-Centric Resource Management in Cloud Computing: A Review and Future Directions](https://mlsanity.com/paper/2105.05079/) (2021)
- [Fast Moving Natural Evolution Strategy for High-Dimensional Problems](https://mlsanity.com/paper/2201.11422/) (2022)
- [Global Optimization: A Machine Learning Approach](https://mlsanity.com/paper/2311.01742/) (2023)
- [Learning Multiple Initial Solutions to Optimization Problems](https://mlsanity.com/paper/2411.02158/) (2024)
- [Same accuracy, twice as fast: continuous training surpasses retraining from scratch](https://mlsanity.com/paper/2502.21147/) (2025)
- [Privacy-Preserving Deep Learning Model for Covid-19 Disease Detection](https://mlsanity.com/paper/2209.04445/) (2022)
- [Learning to Optimize Quasi-Newton Methods](https://mlsanity.com/paper/2210.06171/) (2022)
- [HierSFL: Local Differential Privacy-aided Split Federated Learning in Mobile Edge Computing](https://mlsanity.com/paper/2401.08723/) (2024)

## Abstract (arXiv)

Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this paper, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Next, we summarize the applications and developments of optimization methods in some popular machine learning fields. Finally, we explore and give some challenges and open problems for the optimization in machine learning.