# Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA

arXiv: https://arxiv.org/abs/2610.02067 · HTML: https://mlsanity.com/paper/2610.02067/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning](https://mlsanity.com/paper/2406.09044/) (2024)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors](https://mlsanity.com/paper/2405.19597/) (2024)
- [CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2406.05223/) (2024)
- [Parameter Efficient Fine-tuning via Explained Variance Adaptation](https://mlsanity.com/paper/2410.07170/) (2024)
- [Post-Optimization Adaptive Rank Allocation for LoRA](https://mlsanity.com/paper/2604.27796/) (2026)
- [Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting](https://mlsanity.com/paper/2603.03995/) (2026)
- [Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining](https://mlsanity.com/paper/2605.20296/) (2026)
- [The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning](https://mlsanity.com/paper/2607.23711/) (2026)
- [S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner](https://mlsanity.com/paper/2508.15068/) (2025)

## Abstract (arXiv)

While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify adaptation imbalance: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that learning where to adapt does not ensure that adaptation gains are well balanced. This motivates LoRA-Norm, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.