mlsanityData to 7 Oct 2026

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning

arXiv 2406.18060 ↗First posted 26 Jun 2024cs.CL cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) has achieved remarkable performance across various natural language processing tasks, yet it demands more and more memory as model sizes keep growing. To address this issue, the recently proposed Memory-efficient Zeroth-order (MeZO) methods attempt to fine-tune LLMs using only forward passes, thereby avoiding the need for a backpropagation graph. However, significant performance drops and a high risk of divergence have limited their widespread adoption. In this paper, we propose the Adaptive Zeroth-order Tensor-Train Adaption (AdaZeta) framework, specifically designed to improve the performance and convergence of the ZO methods. To enhance dimension-dependent ZO estimation accuracy, we introduce a fast-forward, low-parameter tensorized adapter. To tackle the frequently observed divergence issue in large-scale ZO fine-tuning tasks, we propose an adaptive query number schedule that guarantees convergence. Detailed theoretical analysis and extensive experimental results on Roberta-Large and Llama-2-7B models substantiate the efficacy of our AdaZeta framework in terms of accuracy, memory efficiency, and convergence speed.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

32 of 46 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
20 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  7. 2014Convex Optimization: Algorithms and Complexitycited by 569
  8. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  10. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  11. 2015Tensorizing Neural Networkscited by 246
  12. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  13. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 75
  14. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  15. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  16. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  17. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 14
  18. 2022BEBERT: Efficient and Robust Binary Ensemble BERTcited by 13
  19. 2022A Bayesian Optimization Framework for Finding Local Optima in Expensive Multi-Modal Functionscited by 12
  20. 2023Positive and Risky Message Assessment for Music Productscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  2. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  3. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Modelscited by 9
  4. 2025Test-Time Model Adaptation for Quantized Neural Networkscited by 9
  5. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  6. 2025ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimizationcited by 4
  7. 2024MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Enginescited by 3
  8. 2025Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimizationcited by 3
  9. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 3
  10. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  11. 2025Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networkscited by 2
  12. 2025Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Modelscited by 2
13 more
  1. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  2. 2025FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregationcited by 1
  3. 2025More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learningcited by 1
  4. 2025On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimizationcited by 1
  5. 2025AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decompositioncited by 0
  6. 2026Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuningcited by 0
  7. 2026Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learningcited by 0
  8. 2026Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNscited by 0
  9. 2026Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapterscited by 0
  10. 2026GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuningcited by 0
  11. 2026Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learningcited by 0
  12. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  13. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 0