mlsanityData to 7 Oct 2026

Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization

arXiv 2605.10673 ↗First posted 11 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Low-bit forward evaluation is an attractive route to memory-efficient zeroth-order (ZO) adaptation: the optimizer needs only scalar losses, and the model can be queried near deployment precision. The obstacle is that a quantized ZO query is not a continuous finite difference followed by harmless storage rounding. The query chooses endpoints, the low-precision engine rounds them, and the loss difference is measured along the rounded chord. For nonuniform companding quantizers, this makes the codebook insufficient to predict ZO behavior: a fixed weight-space radius can collapse in dense cells, over-span sparse cells, or assign a rounded chord to an unrounded update direction. We identify the missing object as query geometry and model scalar nonuniform quantization as Q = φ-1 circ U circ φ. CAQ-ZO (Compander-Aligned Queries for Zeroth-Order Optimization) forms one-grid-step Rademacher stencils z ± Delta r in z = φ(x), maps endpoints back through φ-1, and updates in z. Our theory proves the grid-span mismatch, decomposes endpoint-rounding estimator residuals, and gives stationarity bounds in which generic off-grid queries retain a Delta2/μ2 residual channel while CAQ-ZO makes the query-time residual exactly zero. Synthetic experiments isolate this channel, and matched NF4 Qwen/Llama fine-tuning shows that CAQ-ZO improves the trained NF4 baseline under the same quantizer and evaluation budget.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

19 of 26 references.

References · 19

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  7. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  8. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  9. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  10. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  11. 2015An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedbackcited by 105
  12. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
7 more
  1. 2025Test-Time Model Adaptation for Quantized Neural Networkscited by 9
  2. 2022Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Boundscited by 7
  3. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  4. 2025Fine-tuning Quantized Neural Networks with Zeroth-order Optimizationcited by 5
  5. 2026Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Costcited by 5
  6. 2024Stepping Forward on the Last Milecited by 4
  7. 2025ZOQO: Zero-Order Quantized Optimizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
  2. 2026Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
  3. 2026Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression
  4. 2026FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compression
  5. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
  6. 2026Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
  7. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs
  8. 2026Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference