mlsanityData to 14 Oct 2026

Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning

arXiv 2511.07971 ↗First posted 11 Nov 2025cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach addresses these challenges by: (i) reformulating the problem of gradient preconditioning as that of adaptively estimating an anisotropic perturbation distribution for gradient estimation, (ii) capturing curvature through a low-rank block diagonal preconditioner using the framework of natural evolution strategies, and (iii) applying a REINFORCE leave-one-out (RLOO) gradient estimator to reduce variance. Experiments on standard LLM benchmarks show that our method outperforms state-of-the-art ZO methods by achieving higher accuracy and faster convergence, while cutting peak memory usage by up to 27.3% compared with MeZO-Adam.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

13 of 27 references.

References · 13

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  6. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  7. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  8. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 54
  9. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  10. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  11. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  12. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 14
1 more
  1. 2025MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvaturecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Samplingcited by 2
  2. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  3. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 1
  4. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0
  5. 2026AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuningcited by 0
  6. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0
  7. 2026EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scalecited by 0