# Post-Optimization Adaptive Rank Allocation for LoRA

arXiv: https://arxiv.org/abs/2604.27796 · HTML: https://mlsanity.com/paper/2604.27796/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint](https://mlsanity.com/paper/2509.10971/) (2025)
- [A Simple Approach to Jointly Rank Passages and Select Relevant Sentences in the OBQA Context](https://mlsanity.com/paper/2109.10497/) (2021)

## Built on by (most cited first, 4 of 4)

- [Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation](https://mlsanity.com/paper/2607.02182/) (2026)
- [MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning](https://mlsanity.com/paper/2605.07850/) (2026)
- [Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA](https://mlsanity.com/paper/2610.02067/) (2026)
- [Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding](https://mlsanity.com/paper/2610.12939/) (2026)

## Abstract (arXiv)

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.