mlsanityData to 7 Oct 2026

Improving the Straight-Through Estimator with Zeroth-Order Information

arXiv 2510.23926 ↗First posted 27 Oct 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the problem of training neural networks with quantized parameters. Learning low-precision quantized parameters by enabling computation of gradients via the Straight-Through Estimator (STE) can be challenging. While the STE enables back-propagation, which is a first-order method, recent works have explored the use of zeroth-order (ZO) gradient descent for fine-tuning. We note that the STE provides high-quality biased gradients, and ZO gradients are unbiased but can be expensive. We thus propose First-Order-Guided Zeroth-Order Gradient Descent (FOGZO) that reduces STE bias while reducing computations relative to ZO methods. Empirically, we show FOGZO improves the tradeoff between quality and training time in Quantization-Aware Pre-Training. Specifically, versus STE at the same number of iterations, we show a 1-8% accuracy improvement for DeiT Tiny/Small, 1-2% accuracy improvement on ResNet 18/50, and 1-22 perplexity point improvement for LLaMA models with up to 0.3 billion parameters. For the same loss, FOGZO yields a 796× reduction in computation versus n-SPSA for a 2-layer MLP on MNIST. Code is available at https://github.com/1733116199/fogzo.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

39 of 53 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  10. 2019Learned Step Size Quantizationcited by 604
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
27 more
  1. 2015Deep Learning with Limited Numerical Precisioncited by 368
  2. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  3. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  4. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  5. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  6. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  7. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
  8. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  9. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  10. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  11. 2024Scaling Laws for Precisioncited by 92
  12. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  13. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  14. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  15. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 64
  16. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  17. 2020Universally Quantized Neural Compressioncited by 60
  18. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  19. 2023Training Transformers with 4-bit Integerscited by 58
  20. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  21. 2023A Guide Through the Zoo of Biased SGDcited by 39
  22. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  23. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  24. 2019On Biased Stochastic Gradient Estimationcited by 10
  25. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Modelscited by 9
  26. 2024Custom Gradient Estimators are Straight-Through Estimators in Disguisecited by 6
  27. 2019A Stochastic Gradient Method with Biased Estimation for Faster Nonconvex Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogatescited by 2
  2. 2026Boosting Entropy with Bell Box Quantizationcited by 1
  3. 2026CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Deliverycited by 0
  4. 2026Quantization-Aware Pre-Training with Constrained Empirical Weight Distributioncited by 0
  5. 2026Scaling Zero-Order Pretraining through Model Shardingcited by 0