mlsanityData to 7 Oct 2026

A Survey of Optimization Methods from a Machine Learning Perspective

arXiv 1906.06821 ↗First posted 17 Jun 2019cs.LG math.OC stat.MLcited by 90

Not on the map. Why?

Abstract (arXiv)

Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this paper, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Next, we summarize the applications and developments of optimization methods in some popular machine learning fields. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 21; 2023: 12; 2024: 25; 2025: 12; 2026: 6.

90 papers.

Sort:

By area:

  1. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  2. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  3. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  4. 2022Re-parameterizing Your Optimizers rather than Architecturescited by 35
  5. 2024Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizerscited by 33
  6. 2021Optimization with Constraint Learning: A Framework and Surveycited by 28
  7. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  8. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 23
  9. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  10. 2021HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Searchcited by 20
  11. 2022VectorAdam for Rotation Equivariant Geometry Optimizationcited by 17
  12. 2025Prεεmpt: Sanitizing Sensitive Prompts for LLMscited by 12
38 more
  1. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 10
  2. 2024Generative Adversarial Model-Based Optimization via Source Critic Regularizationcited by 10
  3. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  4. 2021Recommender systems based on graph embedding techniques: A comprehensive reviewcited by 8
  5. 2024A Gaussian Process Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equationscited by 8
  6. 2022Comparative analysis of machine learning methods for active flow controlcited by 6
  7. 2022Unsupervised Tissue Segmentation via Deep Constrained Gaussian Networkcited by 6
  8. 2022Using Human Perception to Regularize Transfer Learningcited by 6
  9. 2022A Continuous Convolutional Trainable Filter for Modelling Unstructured Datacited by 5
  10. 2025Quantum-enhanced Computer Vision: Going Beyond Classical Algorithmscited by 5
  11. 2021Machine Learning (ML)-Centric Resource Management in Cloud Computing: A Review and Future Directionscited by 4
  12. 2022Fast Moving Natural Evolution Strategy for High-Dimensional Problemscited by 4
  13. 2023Global Optimization: A Machine Learning Approachcited by 4
  14. 2024Learning Multiple Initial Solutions to Optimization Problemscited by 4
  15. 2025Same accuracy, twice as fast: continuous training surpasses retraining from scratchcited by 4
  16. 2022Privacy-Preserving Deep Learning Model for Covid-19 Disease Detectioncited by 3
  17. 2022Learning to Optimize Quasi-Newton Methodscited by 3
  18. 2024HierSFL: Local Differential Privacy-aided Split Federated Learning in Mobile Edge Computingcited by 3
  19. 2024Exploiting Memory-aware Q-distribution Prediction for Nuclear Fusion via Modern Hopfield Networkcited by 3
  20. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  21. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  22. 2021Optimal Actor-Critic Policy with Optimized Training Datasetscited by 2
  23. 2022Comparative assessment of federated and centralized machine learningcited by 2
  24. 2022Multi-task Optimization Based Co-training for Electricity Consumption Predictioncited by 2
  25. 2024Mini-Hes: A Parallelizable Second-order Latent Factor Analysis Modelcited by 2
  26. 2024A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Networkcited by 2
  27. 2024You do not have to train Graph Neural Networks at all on text-attributed graphscited by 2
  28. 2025SparseLUT: Sparse Connectivity Optimization for Lookup Table-based Deep Neural Networkscited by 2
  29. 2021Fast symplectic integrator for Nesterov-type acceleration methodcited by 1
  30. 2021A Stochastic Variance-Reduced Coordinate Descent Algorithm for Learning Sparse Bayesian Network from Discrete High-Dimensional Datacited by 1
  31. 2021NEWRON: A New Generalization of the Artificial Neuron to Enhance the Interpretability of Neural Networkscited by 1
  32. 2021Boosting Unsupervised Domain Adaptation with Soft Pseudo-label and Curriculum Learningcited by 1
  33. 2022A Survey and Perspective on Artificial Intelligence for Security-Aware Electronic Design Automationcited by 1
  34. 2022Generalised agent for solving higher board states of tic tac toe using Reinforcement Learningcited by 1
  35. 2023Multiplicative update rules for accelerating deep learning training and increasing robustnesscited by 1
  36. 2024Lightweight Change Detection in Heterogeneous Remote Sensing Images with Online All-Integer Pruning Trainingcited by 1
  37. 2024Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospectscited by 1
  38. 2024Incremental Gauss-Newton Descent for Machine Learningcited by 1

50 most cited of 90 All 90 (JSON)