mlsanityData to 7 Oct 2026

Zeroth order optimization with orthogonal random directions

arXiv 2107.03941 ↗First posted 8 Jul 2021math.OCcited by 21

Not on the map. Why?

Abstract (arXiv)

We propose and analyze a randomized zeroth-order approach based on approximating the exact gradient byfinite differences computed in a set of orthogonal random directions that changes with each iteration. A number ofpreviously proposed methods are recovered as special cases including spherical smoothing, coordinate descent, as wellas discretized gradient descent. Our main contribution is proving convergence guarantees as well as convergence ratesunder different parameter choices and assumptions. In particular, we consider convex objectives, but also possiblynon-convex objectives satisfying the Polyak-Lojasiewicz (PL) condition. Theoretical results are complemented andillustrated by numerical experiments.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jul 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 4; 2025: 9; 2026: 6.

21 papers.

Sort:

  1. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  2. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  3. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  4. 2025Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbationscited by 9
  5. 2022Stochastic Zeroth order Descent with Structured Directionscited by 8
  6. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  7. 2024Forward Learning for Gradient-based Black-box Saliency Map Generationcited by 2
  8. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  9. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  10. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  11. 2022Convergence Rates of Stochastic Zeroth-order Gradient Descent for L ojasiewicz Functionscited by 1
  12. 2024A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detectioncited by 1
9 more
  1. 2024Zeroth-Order Stochastic Mirror Descent Algorithms for Minimax Excess Risk Optimizationcited by 1
  2. 2025Subspace-based Approximate Hessian Method for Zeroth-Order Optimizationcited by 1
  3. 2026Optimal low-rank stochastic gradient estimation for LLM trainingcited by 1
  4. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  5. 2025Stochastic Subspace Descent Accelerated via Bi-fidelity Line Searchcited by 0
  6. 2025A Structured Tour of Optimization with Finite Differencescited by 0
  7. 2026Dimensional Peeking for Low-Variance Gradients in Zeroth-Order Discrete Optimization via Simulationcited by 0
  8. 2026Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNscited by 0
  9. 2026Randomized Subspace Nesterov Accelerated Gradientcited by 0