# GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation

arXiv: https://arxiv.org/abs/2601.00231 · HTML: https://mlsanity.com/paper/2601.00231/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2026.

## Built on by (most cited first, 3 of 3)

- [Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation](https://mlsanity.com/paper/2603.02224/) (2026)
- [Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space](https://mlsanity.com/paper/2603.29824/) (2026)
- [5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2606.10488/) (2026)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) is the default way to adapt LLMs, but widely used LoRA and QLoRA are largely geometry-agnostic: they optimize in fixed, randomly oriented low-rank subspaces with first-order descent, mostly ignoring local loss curvature. This can inflate the effective update budget and amplify drift along weakly constrained directions. We introduce GRIT, a dynamic, curvature-aware LoRA procedure that preserves the LoRA parameterization but: (1) preconditions gradients in rank space using K-FAC as a natural-gradient proxy; (2) periodically reprojects the low-rank basis onto dominant Fisher eigendirections to suppress drift; and (3) adapts the effective rank from the spectrum so capacity concentrates where signal resides. Across instruction-following, comprehension, and reasoning benchmarks on LLaMA backbones, GRIT matches or surpasses LoRA and QLoRA while reducing trainable parameters by 46% on average (25--80% across tasks), without practical quality loss across prompt styles and data mixes. To model forgetting, we fit a curvature-modulated power law. Empirically, GRIT yields lower drift and a better updates-vs-retention frontier than strong PEFT-optimizer baselines (Orthogonal-LoRA, IA3, DoRA, Eff-FT, Shampoo).