mlsanityData to 7 Oct 2026

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer

arXiv 2402.15173 ↗First posted 23 Feb 2024cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) with classic first-order optimizers entails prohibitive GPU memory due to the backpropagation process. Recent works have turned to zeroth-order optimizers for fine-tuning, which save substantial memory by using two forward passes. However, these optimizers are plagued by the heterogeneity of parameter curvatures across different dimensions. In this work, we propose HiZOO, a diagonal Hessian informed zeroth-order optimizer which is the first work to leverage the diagonal Hessian to enhance zeroth-order optimizer for fine-tuning LLMs. What's more, HiZOO avoids the expensive memory cost and only increases one forward pass per step. Extensive experiments on various models (350M~66B parameters) indicate that HiZOO improves model convergence, significantly reducing training steps and effectively enhancing model accuracy. Moreover, we visualize the optimization trajectories of HiZOO on test functions, illustrating its effectiveness in handling heterogeneous curvatures. Lastly, we provide theoretical proofs of convergence for HiZOO. Code is publicly available at https://anonymous.4open.science/r/HiZOO27F8.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 27 of its 57 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

27 of 57 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  12. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
15 more
  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  3. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  4. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  5. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  6. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  7. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  8. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  9. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  10. 2017Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Informationcited by 41
  11. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  12. 2018Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attackcited by 14
  13. 2020A New One-Point Residual-Feedback Oracle For Black-Box Learning and Controlcited by 14
  14. 2018Inexact Non-Convex Newton-Type Methodscited by 11
  15. 2019Gradient-Free Multi-Agent Nonconvex Nonsmooth Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 30.

51 papers.

Sort:

By area:

  1. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  2. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 27
  3. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  4. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 17
  5. 2025FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speedcited by 11
  6. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  7. 2025Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Trainingcited by 9
  8. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  9. 2024HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimizationcited by 7
  10. 2025Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuningcited by 7
  11. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  12. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
38 more
  1. 2025Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizercited by 5
  2. 2025KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuningcited by 4
  3. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  4. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  5. 2025Learning a Zeroth-Order Optimizer for Fine-Tuning LLMscited by 3
  6. 2026CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuningcited by 3
  7. 2026PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMscited by 3
  8. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  9. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  10. 2026AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Momentscited by 2
  11. 2026ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoningcited by 2
  12. 2025Memory Savings at What Cost? A Study of Alternatives to Backpropagationcited by 1
  13. 2025Subspace-based Approximate Hessian Method for Zeroth-Order Optimizationcited by 1
  14. 2025Memory-Efficient Backpropagation for Fine-Tuning LLMs on Resource-Constrained Mobile Devicescited by 1
  15. 2026Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Samplingcited by 1
  16. 2026Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspectivecited by 1
  17. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  18. 2026Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMscited by 1
  19. 2026ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspacescited by 1
  20. 2025ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Modelscited by 0
  21. 2026Robust and Efficient Zeroth-Order LLM Fine-Tuning via Adaptive Bayesian Subspace Optimizercited by 0
  22. 2026Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuningcited by 0
  23. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0
  24. 2026PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Trainingcited by 0
  25. 2026New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Frameworkcited by 0
  26. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  27. 2026Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iterationcited by 0
  28. 2026Inference-Native Zeroth-Order Optimization for LLMscited by 0
  29. 2026Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lenscited by 0
  30. 2026GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuningcited by 0
  31. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0
  32. 2026LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignmentcited by 0
  33. 2026Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptationcited by 0
  34. 2026SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspacescited by 0
  35. 2026EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scalecited by 0
  36. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 0
  37. 2026AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuningcited by 0
  38. 2026Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuningcited by 0

50 most cited of 51 All 51 (JSON)