mlsanityData to 7 Oct 2026

Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models

arXiv 2403.01754 ↗First posted 4 Mar 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient tuning methods such as LoRA could achieve comparable performance to model tuning by tuning a small portion of the parameters. However, substantial computational resources are still required, as this process involves calculating gradients and performing back-propagation throughout the model. Much effort has recently been devoted to utilizing the derivative-free optimization method to eschew the computation of gradients and showcase an augmented level of robustness in few-shot settings. In this paper, we prepend the low-rank modules into each self-attention layer of the model and employ two derivative-free optimization methods to optimize these low-rank modules at each layer alternately. Extensive results on various tasks and language models demonstrate that our proposed method achieves substantial improvement and exhibits clear advantages in memory usage and convergence speed compared to existing gradient-based parameter-efficient tuning and derivative-free optimization methods in few-shot settings.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

40 of 62 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
28 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2021GPT Understands, Toocited by 715
  6. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  12. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  13. 2021True Few-Shot Learning with Language Modelscited by 343
  14. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  15. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  16. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  17. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  18. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  19. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  20. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  21. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  22. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  23. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  24. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  25. 2022Instance-aware Prompt Learning for Language Understanding and Generationcited by 18
  26. 2021Response Generation with Context-Aware Prompt Learningcited by 12
  27. 2015Improving Object Detection with Deep Convolutional Networks via Bayesian Optimization and Structured Predictioncited by 11
  28. 2022Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attentioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 1.

11 papers.

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  3. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  4. 2025Evolution Strategies at the Hyperscalecited by 22
  5. 2024Causal Agent based on Large Language Modelcited by 10
  6. 2025ESSA: Evolutionary Strategies for Scalable Alignmentcited by 6
  7. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  8. 2025EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithmscited by 5
  9. 2024Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approachcited by 1
  10. 2025Using Multi-modal Large Language Model to Boost Fireworks Algorithm's Ability in Settling Challenging Optimization Taskscited by 0
  11. 2026Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuningcited by 0