mlsanityData to 7 Oct 2026

Hyperparameter Optimization for Large Language Model Instruction-Tuning

arXiv 2312.00949 ↗First posted 1 Dec 2023cs.CL math.OCcited by 17

Not on the map. Why?

Abstract (arXiv)

The fine-tuning of Large Language Models (LLMs) has enabled them to recently achieve milestones in natural language processing applications. The emergence of ever larger LLMs has paved the way for more efficient fine-tuning methods. Among these, the Low-Rank Adaptation (LoRA) method keeps most of the weights of the pre-trained LLM frozen while introducing a low-rank decomposition of the weight matrix, enabling the tuning of only a very small proportion of the network. The performance on downstream tasks of models fine-tuned with LoRA heavily relies on a set of hyperparameters including the rank of the decomposition. In this work, we investigate the choice of these hyperparameters through two main blackbox optimization (BBO) techniques. We examine the whole pipeline of performing fine-tuning and validation on a pre-trained LLM as a blackbox and efficiently explore the space of hyperparameters with the nomad algorithm, achieving a boost in performance and human alignment of the tuned model.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

17 of 34 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
5 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  5. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 5; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  2. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  3. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 11
  4. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  5. 2024Pap2Pat: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairscited by 8
  6. 2024Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applicationscited by 6
  7. 2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
  8. 2025HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimizationcited by 4
  9. 2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
  10. 2024ABC Align: Large Language Model Alignment for Safety & Accuracycited by 3
  11. 2025LoRAFusion: Efficient LoRA Fine-Tuning for LLMscited by 3
  12. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
5 more
  1. 2026Automatic Configuration of LLM Post-Training Pipelinescited by 2
  2. 2025CoLoR-GAN: Continual Few-Shot Learning with Low-Rank Adaptation in Generative Adversarial Networkscited by 1
  3. 2026BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Taskscited by 1
  4. 2026Enhancing Language Models for Robust Greenwashing Detectioncited by 0
  5. 2026LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learningcited by 0