# LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

arXiv: https://arxiv.org/abs/2607.19391 · HTML: https://mlsanity.com/paper/2607.19391/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation](https://mlsanity.com/paper/2503.17940/) (2025)
- [Flexora: Flexible Low Rank Adaptation for Large Language Models](https://mlsanity.com/paper/2408.10774/) (2024)
- [ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs](https://mlsanity.com/paper/2506.18267/) (2025)
- [DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models](https://mlsanity.com/paper/2601.04823/) (2026)
- [L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning](https://mlsanity.com/paper/2509.04884/) (2025)
- [FISH-Tuning: Enhancing PEFT Methods with Fisher Information](https://mlsanity.com/paper/2504.04050/) (2025)
- [FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts](https://mlsanity.com/paper/2506.00495/) (2025)
- [FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2412.04416/) (2024)

## Abstract (arXiv)

Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. In this work, we show theoretically and empirically that uniform rank allocation is fundamentally suboptimal. Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training. LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on GLUE and MathInstruct benchmark demonstrate that LAARA consistently matches or outperforms popular state of the art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters. Our results show that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning. The code is publicly available at: https://anonymous.4open.science/r/LAARA-D305/LAARA.py