# BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity

arXiv: https://arxiv.org/abs/2508.06953 · HTML: https://mlsanity.com/paper/2508.06953/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 9 Aug 2025.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [ReLoRA: High-Rank Training Through Low-Rank Updates](https://mlsanity.com/paper/2307.05695/) (2023)
- [LoRA-GA: Low-Rank Adaptation with Gradient Approximation](https://mlsanity.com/paper/2407.05000/) (2024)
- [AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning](https://mlsanity.com/paper/2210.17451/) (2022)
- [KronA: Parameter Efficient Tuning with Kronecker Adapter](https://mlsanity.com/paper/2212.10650/) (2022)
- [HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning](https://mlsanity.com/paper/2404.19245/) (2024)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization](https://mlsanity.com/paper/2305.03937/) (2023)
- [MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2402.12851/) (2024)
- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics](https://mlsanity.com/paper/2505.23194/) (2025)
- [The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning](https://mlsanity.com/paper/2505.23176/) (2025)

## Built on by (most cited first, 4 of 4)

- [Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation](https://mlsanity.com/paper/2510.23123/) (2025)
- [Quantifying Distributional Invariance in Causal Subgraph for IRM-Free Graph Generalization](https://mlsanity.com/paper/2510.20295/) (2025)
- [Diving into Kronecker Adapters: Component Design Matters](https://mlsanity.com/paper/2602.01267/) (2026)
- [SQUARE: Structured Quantum Representation Adapters as Compact Quadratic Feature Maps for Frozen Language Models](https://mlsanity.com/paper/2609.37134/) (2026)

## Abstract (arXiv)

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix W∈ℝm× n by the product of two low-rank matrices, BA, where A ∈ℝr× n and B∈ℝm× r (rllminm,n). Increasing the dimension r can raise the rank of LoRA weights (i.e., BA), which typically improves fine-tuning performance but also significantly increases the number of trainable parameters. In this paper, we propose Block Diversified Low-Rank Adaptation (BoRA), which improves the rank of LoRA weights with a small number of additional parameters. Specifically, BoRA treats the product BA as a block matrix multiplication, where A and B are partitioned into b blocks along the columns and rows, respectively (i.e., A=[A1,dots,Ab] and B=[B1,dots,Bb]^top). Consequently, the product BA becomes the concatenation of the block products BiAj for i,j∈[b]. To enhance the diversity of different block products, BoRA introduces a unique diagonal matrix Sigmai,j ∈ ℝr× r for each block multiplication, resulting in Bi Sigmai,j Aj. By leveraging these block-wise diagonal matrices, BoRA increases the rank of LoRA weights by a factor of b while only requiring b2r additional parameters. Extensive experiments across multiple datasets and models demonstrate the superiority of BoRA, and ablation studies further validate its scalability.