BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix W∈ℝm× n by the product of two low-rank matrices, BA, where A ∈ℝr× n and B∈ℝm× r (rllminm,n). Increasing the dimension r can raise the rank of LoRA weights (i.e., BA), which typically improves fine-tuning performance but also significantly increases the number of trainable parameters. In this paper, we propose Block Diversified Low-Rank Adaptation (BoRA), which improves the rank of LoRA weights with a small number of additional parameters. Specifically, BoRA treats the product BA as a block matrix multiplication, where A and B are partitioned into b blocks along the columns and rows, respectively (i.e., A=[A1,dots,Ab] and B=[B1,dots,Bb]^top). Consequently, the product BA becomes the concatenation of the block products BiAj for i,j∈[b]. To enhance the diversity of different block products, BoRA introduces a unique diagonal matrix Sigmai,j ∈ ℝr× r for each block multiplication, resulting in Bi Sigmai,j Aj. By leveraging these block-wise diagonal matrices, BoRA increases the rank of LoRA weights by a factor of b while only requiring b2r additional parameters. Extensive experiments across multiple datasets and models demonstrate the superiority of BoRA, and ablation studies further validate its scalability.