mlsanityData to 14 Oct 2026

PoLAR: Polar-Decomposed Low-Rank Adapter Representation

arXiv 2506.03133 ↗First posted 3 Jun 2025cs.LG cs.AI eess.SPcited by 20

Not on the map. Why?

Abstract (arXiv)

We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 74 of its 101 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

74 of 101 references.

References · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  13. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  14. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  15. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  16. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  17. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  18. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  19. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  20. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  21. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  22. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  23. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  24. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  25. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  26. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  27. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  28. 2013Manopt, a Matlab toolbox for optimization on manifoldscited by 113
  29. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  30. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  31. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  32. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  33. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  34. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  35. 2018Learning towards Minimum Hyperspherical Energycited by 75
  36. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  37. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  38. 2023Transformers learn through gradual rank increasecited by 52

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 15.

20 papers.

Sort:

By area:

  1. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 14
  2. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  3. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 5
  4. 2025Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFTcited by 4
  5. 2025On the Benefits of Weight Normalization for Overparameterized Matrix Sensingcited by 3
  6. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  7. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  8. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  9. 2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
  10. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  11. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuningcited by 1
  12. 2026SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inferencecited by 1
8 more
  1. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 1
  2. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  3. 2026Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamicscited by 0
  4. 2026Muon on the Stiefel Manifold Admits an Exact Closed-Form Updatecited by 0
  5. 2026Behaviorally Effective LoRA Writes Are Sparse and Structuredcited by 0
  6. 2026Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descentcited by 0
  7. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  8. 2026LoRA's Second Descent Extends Beyond Parameter Paritycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rotated Manifold Optimization for Low-Rank Adaptation
  2. 2025AROMA: Autonomous Rank-one Matrix Adaptation
  3. 2026Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
  4. 2025Mitigating Forgetting in Low Rank Adaptation
  5. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  6. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  7. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  8. 2025SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model