mlsanityData to 14 Oct 2026

Normalized Low-Rank Adaptation

arXiv 2608.31036 ↗First posted 31 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 66 of its 76 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

66 of 76 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2023Let's Verify Step by Stepcited by 3,789
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  13. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  14. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  15. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  16. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  17. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  18. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  19. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  20. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  21. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  22. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  23. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  24. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  25. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  26. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  27. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  28. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  29. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  30. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  31. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  32. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  33. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  34. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  35. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  36. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  37. 2022Neural Prompt Searchcited by 135
  38. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133

50 most cited of 66 All 66 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rotated Manifold Optimization for Low-Rank Adaptation
  2. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
  3. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
  4. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  5. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning
  6. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  7. 2023The Expressive Power of Low-Rank Adaptation
  8. 2025AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing