mlsanityData to 7 Oct 2026

A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order

arXiv 1606.00498 ↗First posted 1 Jun 2016math.OCcited by 43

Not on the map. Why?

Abstract (arXiv)

Asynchronous parallel optimization received substantial successes and extensive attention recently. One of core theoretical questions is how much speedup (or benefit) the asynchronous parallelization can bring us. This paper provides a comprehensive and generic analysis to study the speedup property for a broad range of asynchronous parallel stochastic algorithms from the zeroth order to the first order methods. Our result recovers or improves existing analysis on special cases, provides more insights for understanding the asynchronous parallel behaviors, and suggests a novel asynchronous parallel zeroth order method for the first time. Our experiments provide novel applications including model blending problems using the proposed asynchronous parallel zeroth order method.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 6; 2023: 6; 2024: 4; 2025: 12; 2026: 4.

43 papers.

Sort:

By area:

  1. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  2. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  3. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  4. 2022Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimizationcited by 40
  5. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  6. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  7. 2021L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimizationcited by 21
  8. 2021Distributed Learning Systems with First-order Methodscited by 17
  9. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  10. 2021Optimizing Large-Scale Hyperparameters via Automated Learning Algorithmcited by 13
  11. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
  12. 2023Certified Zeroth-order Black-Box Defense with Robust UNet Denoisercited by 11
31 more
  1. 2025Refining Adaptive Zeroth-Order Optimization at Easecited by 11
  2. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  3. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  4. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  5. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  6. 2025Zeroth-Order Optimization is Secretly Single-Step Policy Optimizationcited by 7
  7. 2021Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithmcited by 6
  8. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  9. 2023STARNet: Sensor Trustworthiness and Anomaly Recognition via Approximated Likelihood Regret for Robust Edge Autonomycited by 5
  10. 2021SGD with Coordinate Sampling: Theory and Practicecited by 4
  11. 2025ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimizationcited by 4
  12. 2025The Multi-Query Paradox in Zeroth-Order Optimizationcited by 4
  13. 2021Distributed stochastic optimization with large delayscited by 3
  14. 2022Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivitycited by 3
  15. 2025Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networkscited by 2
  16. 2021Convergence Analysis of Nonconvex Distributed Stochastic Zeroth-order Coordinate Methodcited by 1
  17. 2021An Accelerated Variance-Reduced Conditional Gradient Sliding Algorithm for First-order and Zeroth-order Optimizationcited by 1
  18. 2022Adaptive Zeroth-Order Optimisation of Nonconvex Composite Objectivescited by 1
  19. 2024Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradientcited by 1
  20. 2024Single Point-Based Distributed Zeroth-Order Optimization with a Non-Convex Stochastic Objective Functioncited by 1
  21. 2025Adversarial Attacks on Downstream Weather Forecasting Models: Application to Tropical Cyclone Trajectory Predictioncited by 1
  22. 2026ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimizationcited by 1
  23. 2021Learning Sampling Policy for Faster Derivative Free Optimizationcited by 0
  24. 2022An Empirical Evaluation of Zeroth-Order Optimization Methods on AI-driven Molecule Optimizationcited by 0
  25. 2022Adaptive Stochastic Optimisation of Nonconvex Composite Objectivescited by 0
  26. 2024Obtaining Lower Query Complexities through Lightweight Zeroth-Order Proximal Gradient Algorithmscited by 0
  27. 2025VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergencecited by 0
  28. 2025Membership Inference Attack with Partial Featurescited by 0
  29. 2026Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimatescited by 0
  30. 2026Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lenscited by 0
  31. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0