mlsanityData to 14 Oct 2026

Hierarchical Zero-Order Optimization for Deep Neural Networks

arXiv 2602.10607 ↗First posted 11 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization has long been favored for its biological plausibility and its capacity to handle non-differentiable objectives, yet its computational complexity has historically limited its application in deep neural networks. Challenging the conventional paradigm that gradients propagate layer-by-layer, we propose Hierarchical Zeroth-Order (HZO) optimization, a novel divide-and-conquer strategy that decomposes the depth dimension of the network. We prove that HZO reduces the query complexity from O(ML2) to O(ML log L) for a network of width M and depth L, representing a significant leap over existing ZO methodologies. Furthermore, we provide a detailed error analysis showing that HZO maintains numerical stability by operating near the unitary limit (Llip ≈ 1). Extensive evaluations on CIFAR-10 and ImageNet demonstrate that HZO achieves competitive accuracy compared to backpropagation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

14 of 29 references.

References · 14

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  3. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  4. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 79
  5. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  6. 2022Scaling Forward Gradient With Local Lossescited by 65
  7. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 56
  8. 2020Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimizationcited by 24
  9. 2022Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathwayscited by 14
  10. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Modelscited by 11
  11. 2023Training neural networks with end-to-end optical backpropagationcited by 7
  12. 2024An Accelerated Gradient Method for Convex Smooth Simple Bilevel Optimizationcited by 4
2 more
  1. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  2. 2024Hidden semi-Markov models with inhomogeneous state dwell-time distributionscited by 1

Built on by

Not cited yet.