A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search
Not on the map. Why?
Abstract (arXiv)
Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) offers a resource-efficient way to personalize or specialize. However, LoRA is highly sensitive to hyperparameter choices, and exhaustive hyperparameter search is computationally expensive. To address this, we propose a Bayesian Optimization (BO) framework that leverages the domain knowledge of pre-trained LLMs to efficiently search for LoRA hyperparameters. Our approach repurposes a pre-trained LLM as a discrete-to-continuous mapping module to link hyperparameters and their domain knowledge to a continuous vector space, where BO is conducted. We design and control the mapping via language prompting, providing a domain-aware textual prompt that describes the relationships among hyperparameters and their respective roles. This allows us to explicitly inject domain knowledge about LoRA into the LLM in natural language. We also introduce an additional learnable token to capture residual information that is difficult to describe linguistically in the prompt. This aids BO to sample more high-performing hyperparameters. In addition, by leveraging the strong correlation observed between the performance obtained from full and subset training datasets in LoRA training regimes, we introduce proxy training and evaluation using a data subset. This significantly improves the efficiency of our method. We demonstrate that our hyperparameter, discovered with only about 30 iterations, achieves more than 20% performance improvement over standard hyperparameters found from about 45,000 combinations. Project page: https://baekseongeun.github.io/lora-bo/
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 49 of its 73 references that are in the map. First posted on arXiv 19 Jan 2026.
Builds on
49 of 73 references.
References · 49
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2024Qwen2 Technical Reportcited by 2,076
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
37 more
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
- 2015Deep Kernel Learningcited by 440
- 2024LoRA Learns Less and Forgets Lesscited by 313
- 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
- 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
- 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
- 2024Large Language Models to Enhance Bayesian Optimizationcited by 127
- 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
- 2019Combinatorial Bayesian Optimization using the Graph Cartesian Productcited by 78
- 2019ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendationscited by 77
- 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
- 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
- 2024A Survey on LoRA of Large Language Modelscited by 67
- 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 62
- 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
- 2021Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spacescited by 45
- 2018High Dimensional Bayesian Optimization Using Dropoutcited by 40
- 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
- 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
- 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 30
- 2024TSDS: Data Selection for Task-Specific Model Finetuningcited by 28
- 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
- 2025Latent Bayesian Optimization via Autoregressive Normalizing Flowscited by 19
- 2023Hyperparameter Optimization for Large Language Model Instruction-Tuningcited by 17
- 2024Inversion-based Latent Bayesian Optimizationcited by 11
- 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
- 2025RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptationcited by 9
- 2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gatescited by 8
- 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
- 2024Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applicationscited by 6
- 2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
- 2025PLoRA: Efficient Concurrent LoRA Training for Large Language Modelscited by 6
- 2025How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustnesscited by 4
- 2021An Introduction to Gaussian Process Modelscited by 3
- 2024Parameter Efficient Instruction Tuning: An Empirical Studycited by 3
- 2024Crafting Efficient Fine-Tuning Strategies for Large Language Modelscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.