mlsanityData to 14 Oct 2026

Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA

arXiv 2610.06621 ↗First posted 5 Oct 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B also trails comparable-budget free LoRA by 8-18 percentage points on five pairs spanning a formatting task and OpenBookQA. The A-frozen advantage persists in a single-GPU-model replication and within individual MLP module groups, including controls with equal or greater trainable counts for B frozen, and when A is frozen on a random orthonormal basis. The factor contrast weakens with rank. On OpenBookQA / Qwen2.5-1.5B at rank 16, PEFT's MiCA implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. A trained oracle output subspace largely removes the low-rank deficit; partial warm-up gains recur across three direction seeds. The factor-versus-band ordering is descriptive; an approximate multiplicity audit weakens several earlier significance claims. These results extend known factor asymmetry by showing how its magnitude depends on spectral placement, rank and training conditions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

30 of 34 references.

References · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2023Mistral 7Bcited by 3,164
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  10. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  11. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  12. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
18 more
  1. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  2. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  3. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  4. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  6. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  7. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  8. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  9. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 38
  10. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  11. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  12. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 11
  13. 2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 6
  14. 2025SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASRcited by 4
  15. 2026Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRAcited by 4
  16. 2026Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offscited by 2
  17. 2026MiCA Learns More Knowledge Than LoRA and Full Fine-Tuningcited by 1
  18. 2026Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learningcited by 1

Built on by

Not cited yet.