mlsanityData to 14 Oct 2026

ZOQO: Zero-Order Quantized Optimization

arXiv 2501.06736 ↗First posted 12 Jan 2025cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 12 Jan 2025.

Builds on

31 of 39 references.

References · 31

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  7. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  10. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  11. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  12. 2019Simple Black-box Adversarial Attackscited by 251
19 more
  1. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  2. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  3. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  4. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  5. 2018AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networkscited by 137
  6. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  7. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  8. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  9. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  10. 2017Stochastic Zeroth-order Optimization in High Dimensionscited by 63
  11. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 54
  12. 2020Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Samplingcited by 46
  13. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  14. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  15. 2018Zeroth-order Nonconvex Stochastic Optimization: Handling Constraints, High-Dimensionality and Saddle-Pointscited by 37
  16. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 28
  17. 2020Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimizationcited by 24
  18. 2021On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithmscited by 23
  19. 2019On the Design of Black-box Adversarial Examples by Leveraging Gradient-free Optimization and Operator Splitting Methodcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Fine-tuning Quantized Neural Networks with Zeroth-order Optimizationcited by 6
  2. 2026Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimizationcited by 0
  3. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  4. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving the Straight-Through Estimator with Zeroth-Order Information
  2. 2025Zero-shot Quantization: A Comprehensive Survey
  3. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
  4. 2021Zero-shot Adversarial Quantization
  5. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
  6. 2023Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
  7. 2022It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher
  8. 2026SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuning