# BoostLoRA: Growing Effective Rank by Boosting Adapters

arXiv: https://arxiv.org/abs/2604.27308 · HTML: https://mlsanity.com/paper/2604.27308/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [ReLoRA: High-Rank Training Through Low-Rank Updates](https://mlsanity.com/paper/2307.05695/) (2023)
- [Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning](https://mlsanity.com/paper/2401.04151/) (2024)
- [StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback](https://mlsanity.com/paper/2402.01391/) (2024)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks](https://mlsanity.com/paper/2410.13025/) (2024)
- [Learning Deep ResNet Blocks Sequentially using Boosting Theory](https://mlsanity.com/paper/1706.04964/) (2017)
- [Multi-Turn Code Generation Through Single-Step Rewards](https://mlsanity.com/paper/2502.20380/) (2025)
- [MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2402.17263/) (2024)
- [Learning to Reason in 13 Parameters](https://mlsanity.com/paper/2602.04118/) (2026)
- [Evaluating Parameter Efficient Methods for RLVR](https://mlsanity.com/paper/2512.23165/) (2025)
- [Beyond Simple Concatenation: Fairly Assessing PLM Architectures for Multi-Chain Protein-Protein Interactions Prediction](https://mlsanity.com/paper/2505.20036/) (2025)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) methods face a tradeoff between adapter size and expressivity: ultra-low-parameter adapters are confined to fixed low-rank subspaces, capping performance even with extended training. We propose BoostLoRA, a gradient-boosting framework that overcomes this limit by iteratively training and merging minimal adapters on the examples the current model gets wrong. A ROTATE SVD basis strategy assigns each round to an orthogonal subspace, so cumulative effective rank grows linearly with the number of rounds while each adapter remains ultra-low-rank. After merging, adapters are discarded, leaving zero inference overhead. On Qwen2.5-3B, BoostLoRA reaches 89.1% on GSM8K and 68.8% on MATH-500, surpassing both the best single-shot ultra-low parameter adapter (TinyLoRA) and full fine-tuning; on code generation it reaches 57.2% on MBPP and 80.4% on HumanEval while full fine-tuning drops below the zero-shot baseline. We also demonstrate cross-architecture transfer on protein binding classification with ESM2-650M and cross-entropy training. BoostLoRA is, to our knowledge, the first PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity.