mlsanityData to 14 Oct 2026

Fine-Tuning Language Models with Just Forward Passes

arXiv 2305.17333 ↗First posted 27 May 2023cs.LG cs.CLcited by 345

Not on the map. Why?

Abstract (arXiv)

Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients using only two forward passes but are theorized to be catastrophically slow for optimizing large models. In this work, we propose a memory-efficient zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate in-place, thereby fine-tuning LMs with the same memory footprint as inference. For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter model, whereas fine-tuning with backpropagation can train only a 2.7B LM with the same budget. We conduct comprehensive experiments across model types (masked and autoregressive LMs), model scales (up to 66B), and downstream tasks (classification, multiple-choice, and generation). Our results demonstrate that (1) MeZO significantly outperforms in-context learning and linear probing; (2) MeZO achieves comparable performance to fine-tuning with backpropagation across multiple tasks, with up to 12x memory reduction and up to 2x GPU-hour reduction in our implementation; (3) MeZO is compatible with both full-parameter and parameter-efficient tuning techniques such as LoRA and prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives (e.g., maximizing accuracy or F1). We support our empirical findings with theoretical insights, highlighting how adequate pre-training and task prompts enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting otherwise.

Key facts. Cited by 345 papers in the map (data through 14 Oct 2026). Builds on 79 of its 116 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

79 of 116 references.

References · 79

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  13. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  14. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  15. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  16. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  17. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  18. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  19. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  20. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  21. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  22. 2017Large Batch Training of Convolutional Networkscited by 535
  23. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  24. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  25. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  26. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  27. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  28. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  29. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  30. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  31. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  32. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  33. 2020Understanding the Difficulty of Training Transformerscited by 210
  34. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  35. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  36. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  37. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  38. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 32; 2024: 92; 2025: 104; 2026: 117.

345 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  4. 2023Challenges and Applications of Large Language Modelscited by 254
  5. 2023Efficient Large Language Models: A Surveycited by 186
  6. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  7. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  9. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  10. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  11. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  12. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
38 more
  1. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  2. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  4. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  5. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  6. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76
  7. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  8. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 71
  9. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  10. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  11. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  12. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
  13. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  14. 2023Learning from models beyond fine-tuningcited by 48
  15. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  16. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 43
  17. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  18. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  19. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  20. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  21. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 31
  22. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  23. 2024The Fine-Grained Complexity of Gradient Computation for Training Large Language Modelscited by 30
  24. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 28
  25. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  26. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  27. 2025RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memorycited by 27
  28. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  29. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 26
  30. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  31. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 26
  32. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 25
  33. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  34. 2025Evolution Strategies at the Hyperscalecited by 25
  35. 2023An Algorithm with Optimal Dimension-Dependence for Zero-Order Nonsmooth Nonconvex Stochastic Optimizationcited by 23
  36. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 22
  37. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  38. 2024Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operatorscited by 20

50 most cited of 345 All 345 (JSON)