mlsanityData to 14 Oct 2026

ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization

arXiv 2501.04287 ↗First posted 8 Jan 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization is being recognized as a simple yet powerful alternative to standard backpropagation (BP)-based training. Notably, ZO optimization allows for training with only forward passes and (almost) the same memory as inference, making it well-suited for edge devices with limited computing and memory resources. In this paper, we propose ZO-based on-device learning (ODL) methods for full-precision and 8-bit quantized deep neural networks (DNNs), namely ElasticZO and ElasticZO-INT8. ElasticZO lies in the middle between pure ZO- and pure BP-based approaches, and is based on the idea to employ BP for the last few layers and ZO for the remaining layers. ElasticZO-INT8 achieves integer arithmetic-only ZO-based training for the first time, by incorporating a novel method for computing quantized ZO gradients from integer cross-entropy loss values. Experimental results on the classification datasets show that ElasticZO effectively addresses the slow convergence of vanilla ZO and shrinks the accuracy gap to BP-based training. Compared to vanilla ZO, ElasticZO achieves 5.2-9.5% higher accuracy with only 0.072-1.7% memory overhead, and can handle fine-tuning tasks as well as full training. ElasticZO-INT8 further reduces the memory usage and training time by 1.46-1.60x and 1.38-1.42x without compromising the accuracy. These results demonstrate a better tradeoff between accuracy and training cost compared to pure ZO- and BP-based approaches, and also highlight the potential of ZO optimization in on-device learning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

35 of 51 references.

References · 35

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  6. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  7. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  8. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  9. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  10. 2018AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networkscited by 137
  11. 2020Efficient Domain Generalization via Common-Specific Low-Rank Decompositioncited by 134
  12. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
23 more
  1. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  2. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  3. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  4. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  5. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  6. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 54
  7. 2019Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimizationcited by 47
  8. 2016A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Ordercited by 44
  9. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  10. 2020Accelerated Zeroth-Order and First-Order Momentum Methods from Mini to Minimax Optimizationcited by 40
  11. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  12. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35
  13. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 28
  14. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 25
  15. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  16. 2017Zeroth-Order Online Alternating Direction Method of Multipliers: Convergence Analysis and Applicationscited by 22
  17. 2020NITI: Training Integer Neural Networks Using Integer-only Arithmeticcited by 20
  18. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 19
  19. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 18
  20. 2024PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMscited by 18
  21. 2018Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attackcited by 14
  22. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  23. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  2. 2025Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networkscited by 2
  3. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  4. 2026Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theorycited by 0
  5. 2026Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ZOQO: Zero-Order Quantized Optimization
  2. 2025Improving the Straight-Through Estimator with Zeroth-Order Information
  3. 2025On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
  4. 2025FF-INT8: Efficient Forward-Forward DNN Training on Edge Devices with INT8 Precision
  5. 2026QScheduler: Adaptive Gradient Sampling for Zeroth-Order On-Device Training on INT8 NPUs
  6. 2021Enabling Binary Neural Network Training on the Edge
  7. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
  8. 2025Fine-tuning Quantized Neural Networks with Zeroth-order Optimization