BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes

Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. However, the link between fine-tuning different bias terms (i.e., boldsymbolbq, boldsymbolbk, and boldsymbolbv in the query, key, or value projections) and downstream performance remains largely unclear to date. In this paper, we investigate the link between fine-tuning boldsymbolbq, boldsymbolbk, and boldsymbolbv with the performance of the downstream task. Our key finding is that directly fine-tuning boldsymbolbv generally leads to higher downstream performance in low-data regimes, in comparison to boldsymbolbq and boldsymbolbk. We extensively evaluate this unique property across a wide range of LLMs spanning encoder-only and decoder-only architectures up to 6.7B parameters (including bias-free LLMs). Our results provide strong evidence for the effectiveness of directly fine-tuning boldsymbolbv across various downstream tasks. The implementation code is available at https://github.com/whubaichuan/BEFT.