mlsanityData to 14 Oct 2026

Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training

arXiv 2504.20314 ↗First posted 28 Apr 2025cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization is an emerging deep neural network (DNN) training paradigm that offers computational simplicity and memory savings. However, this seemingly promising approach faces a significant and long-ignored challenge. ZO requires generating a substantial number of Gaussian random numbers, which poses significant difficulties and even makes it infeasible for hardware platforms, such as FPGAs and ASICs. In this paper, we identify this critical issue, which arises from the mismatch between algorithm and hardware designers. To address this issue, we proposed PeZO, a perturbation-efficient ZO framework. Specifically, we design random number reuse strategies to significantly reduce the demand for random number generation and introduce a hardware-friendly adaptive scaling method to replace the costly Gaussian distribution with a uniform distribution. Our experiments show that PeZO reduces the required LUTs and FFs for random number generation by 48.6% and 12.7%, and saves at maximum 86% power consumption, all without compromising training performance, making ZO optimization feasible for on-device training. To the best of our knowledge, we are the first to explore the potential of on-device ZO optimization, providing valuable insights for future research.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

24 of 47 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  7. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  8. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  9. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  10. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  11. 2015An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedbackcited by 111
  12. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
12 more
  1. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
  2. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  3. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  4. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  5. 2019Model Agnostic Contrastive Explanations for Structured Datacited by 41
  6. 2020Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimizationcited by 24
  7. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 18
  8. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 18
  9. 2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptationcited by 16
  10. 2020Learning to Guide Random Searchcited by 13
  11. 2023Certified Zeroth-order Black-Box Defense with Robust UNet Denoisercited by 11
  12. 2018FPGA-Based CNN Inference Accelerator Synthesized from Multi-Threaded C Softwarecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memorycited by 27
  2. 2025Structured Agent Distillation for Large Language Modelcited by 13
  3. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  4. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Costcited by 3
  5. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  6. 2025TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platformcited by 1
  7. 2026Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspectivecited by 1
  8. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  9. 2026Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformerscited by 0